LSM data organization method and apparatus for sparse metadata distribution

Abstract

Disclosed in the present invention are an LSM data organization method and apparatus for sparse metadata distribution. The present invention uses dynamic partitioning to organize sorted string tables. Specifically, clustering analysis is performed on key names of key-value pair data first to obtain key-name groups, then the key-name groups are all pruned to form key-value pair partitions, and the key-value pair data is flushed into the corresponding key-value pair partitions to form the sorted string tables. Each key-value pair partition independently performs a merge-sort operation, such that the forming of sorted string tables with sparse metadata distribution is effectively limited, thus greatly reducing the number of high-level sorted string tables participating in the merge-sort process, and reducing the frequency of the high-level sorted string tables repeatedly participating in the merge-sort operation. During retrieval of the key-value pair data, the key-value pair partitions further accelerate the locating of the key-value pair data, thereby improving the read-write performance of a system.

Description

A LSM data organization method and device for sparsely distributed metadata Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular relates to a log structured merge (LSM) data organization method and device for sparsely distributed metadata. Background Art

[0002] LSM-based databases implement random reads and writes of massive key-value pairs through random reads and writes in memory, sequential flushing to disk, and periodic merging. This leverages the fact that sequential writes on disk outperform random writes. When the in-memory Mutable MemTable (hereinafter referred to as the mutable memtable) reaches a threshold, it is converted to the Immutable MemTable (hereinafter referred to as the immutable memtable) state and subsequently flushed to disk.

[0003] When the key-value pair data is flushed to disk, it is first flushed to form a sorted string table and placed in the lower level. Then, with the merge sort operation, the key-value pair data is migrated upward layer by layer. Among them, the merge sort operation will first select a sorted string table from the lower level as the low-level sorted string table input, and then select a sorted string table in the adjacent higher level whose key control range intersects with the key control range of the lower level sorted string table, and use it as the higher-level sorted string table input. The merge sort of the key-value pair data is completed in memory and flushed to the higher level again. If the key-value pair data writing speed is faster than the merge sort operation speed, the lower level will be delayed in completing the sorted string table migration, delaying the release of space, and thus blocking the write. Therefore, the speed of the merge sort operation directly affects the write speed of the system. However, when faced with a low-level sorting string table with sparsely distributed metadata, that is, the low-level sorting string table contains fewer key-value pair data, but the key names of the key-value pair data are distributed over a wide range, in this case, the small-volume low-level sorting string table will cause a large number of high-level sorting string tables to participate in the merge sort operation. As the level increases, the number of high-level sorting string tables that repeatedly participate in the merge sort process will increase and the volume will be larger, which will occupy a large amount of disk bandwidth, CPU and other system resources, greatly affecting the system's read and write performance.

[0004] Summary of the Invention

[0005] In response to the problems existing in the prior art, the purpose of the present invention is to provide an LSM data organization method and device for sparsely distributed metadata, so as to improve the read and write performance of LSM-based databases and alleviate the problem in the prior art that the sparsely distributed metadata sorting string table under the LSM data structure causes a large number of high-level sorting string tables to repeatedly participate in the merge sort process, occupying a large amount of system resources and reducing the system read and write performance.

[0006] The present invention uses dynamic partitioning to organize the sorted string table. Specifically, the key names of the key-value pair data are first clustered and analyzed to obtain key name groups. The key name groups are then pruned to form key-value pair partitions. The key-value pair data are flushed into the corresponding key-value pair partitions to form a sorted string table. Each key-value pair partition independently performs a merge sort operation, thereby effectively limiting the formation of a sorted string table with sparsely distributed metadata, thereby greatly reducing the number of high-level sorted string tables participating in the merge sort process, and reducing the number of times the high-level sorted string tables repeatedly participate in the merge sort operation. When retrieving key-value pair data, the key-value pair partitions further speed up the positioning of the key-value pair data, thereby achieving an improvement in the read and write performance of the system.

[0007] The specific technical solutions provided by the present invention are as follows:

[0008] First, an LSM data organization method for sparsely distributed metadata is proposed, including the following steps:

[0009] Receive written key-value pair data through the mutable memory table in memory;

[0010] When the variable memory table reaches a threshold, convert it into one or more immutable memory tables;

[0011] Writing the key-value pair data in the plurality of immutable memory tables to disk;

[0012] When the first trigger is triggered to convert the mutable memtable into an immutable memtable:

[0013] Perform cluster analysis on the key names of the key-value pair data in the immutable memory table based on the key name distance to obtain several key name groups;

[0014] Pruning each of the key name groups to obtain a key-value pair partition, and writing the key-value pair data in the key-value pair partition to form a sorted string table;

[0015] A partition pre-divided in the memory is used as a temporary storage area for key-value pair data; the key-value pair data outside the key-value pair partition is uniformly stored in the temporary storage area for key-value pair data in the memory area;

[0016] When converting a mutable memtable to an immutable memtable for the first time:

[0017] All key-value pair partitions are listed in order of key control range size, and information of the key-value pair partitions is obtained, including: the key control range of the key-value pair partition is recorded as [A, B], the expansion coefficient of the key-value pair partition is recorded as T, the key control range size threshold of the key-value pair partition is recorded as R, etc.;

[0018] Retrieve the key-value data in the immutable memory table, filter out the key-value data with the key name in the interval [AT, B+T], write it to the corresponding key-value partition to form a sorted string table, and update the key control range and other information of the key-value partition;

[0019] Obtain the key names of the key-value pair data remaining after the above screening and the key-value pair data in the key-value pair data temporary storage area, and perform a cluster analysis operation to obtain a new key name group again;

[0020] A new key-value pair partition is obtained by pruning the new key name group, and the key-value pair data in the new key-value pair partition is flushed to form a sorted string table.

[0021] Furthermore, the left boundary value of the key control range of the key-value pair partition is the minimum key in the key name group, and the right boundary value of the key control range of the key-value pair partition is the maximum key in the key name group. If the absolute difference between the right boundary value and the left boundary value of the key control range of the key-value pair partition exceeds the key control range size threshold R of the key-value pair partition, the key control range of the key-value pair partition is pruned according to the key control range size threshold R of the key-value pair partition. The pruning method is to eliminate the minimum key in sequence according to the key size until the absolute difference between the right boundary value and the left boundary value of the key control range of the key-value pair partition remains within the key control range size threshold R of the key-value pair partition.

[0022] Furthermore, not only is it necessary to perform the above processing so that the absolute difference between the right boundary value and the left boundary value of the key range of the new key-value pair partition remains within the key range size threshold R of the key-value pair partition, but it is also necessary to ensure that there is no intersection between the key ranges of the new key-value pair partition and the key ranges of the existing key-value pair partitions. The key range size threshold R of each key-value pair partition is the same.

[0023] Furthermore, the key-value data temporary storage area is implemented using an efficient retrieval data structure, such as a hash table. There is no need to consider the scenario of frequent element updates, and it only needs to support fast retrieval. At the same time, the key-value data temporary storage area needs to provide the external module with an interface F for quickly obtaining the full amount of key-value pair data in the temporary storage area and an interface G for obtaining the full amount of key names in the temporary storage area. The F interface only needs to return a list of all key-value pair data sorted by key names stored in the temporary storage area, and the G interface only needs to return a list of all key names stored in the temporary storage area.

[0024] Furthermore, the left and right boundary values ​​of the interval [AT, B+T] need to be adjusted as needed. If the key-value pair partition i where the key name of the currently filtered key-value pair is located has a previous key-value pair partition j, then the left boundary value AT of the interval needs to be greater than the right boundary value of the key control range of the key-value pair partition j. If the key-value pair partition i has a subsequent key-value pair partition k, then the right boundary value B+T of the interval needs to be less than the left boundary value of the key control range of the key-value pair partition k, and the size of the interval B-A+2*T needs to be kept within the key control range size threshold R of the key-value pair partition.

[0025] Secondly, an LSM data organization device for sparsely distributed metadata is proposed, including:

[0026] A key name acquisition module is used to obtain the key names of key-value pairs in an immutable memory table, or to obtain the key names of a given batch of key-value pairs, including obtaining the key names of key-value pairs in an immutable memory table, obtaining the key names of the remaining key-value pairs after filtering, and obtaining the key names of key-value pairs in a temporary storage area.

[0027] The key name clustering analysis module is used to perform cluster analysis on batch key names to obtain key name groups, and further process them to obtain key-value pair partitions. Specifically, the key name groups will be pruned according to the key control range size threshold R of the key-value pair partition and the existing key-value pair partitions to ensure that: 1) the absolute difference between the right boundary value and the left boundary value of the key control range of the key-value pair partition remains within the key control range size threshold R of the key-value pair partition; 2) the key control ranges of the key-value pair partition and the existing key-value pair partitions do not intersect. At the same time, the module supports the definition of key name distance metrics that meet business needs and supports the configuration of efficient and accurate clustering analysis algorithms.

[0028] The key-value partition management module is used to store and manage key-value partition information, including key control range, expansion coefficient, key control range size threshold, number of sorted string tables, etc. At the same time, it supports various operations on the key-value partitions, including adding new key-value partitions, updating key-value partition information, and listing key-value partitions.

[0029] A key-value data temporary storage area module is used to store key-value data outside the key name group during system operation. This module is located in the memory and mainly consists of a data structure that supports efficient retrieval. At the same time, the module is configured with a related logging mechanism to ensure the consistency, persistence and reliability of the key-value data in the module;

[0030] The key-value data writing module is used to coordinate the key-value data writing process. First, it obtains the key-value partition list from the key-value partition management module. If the key-value partition list is empty, it obtains the key names of all key-value data in the immutable memory table from the key name acquisition module, and transmits them to the key-value clustering analysis module for clustering analysis to obtain key-value partitions. It writes the key-value data in the key-value partitions and stores the key-value data outside the key-value partitions in the key-value data temporary storage area module. At the same time, the key-value partition management module manages the key-value data. For partitions, if the key-value pair partition list is not empty, the key-value pair data is flushed to the corresponding key-value pair partition to form a sorted string table, the remaining key-value pair data and the key name of the key-value pair data in the key-value pair data temporary storage area module are transmitted to the key-value pair clustering analysis module for clustering analysis to obtain a new key-value pair partition, and the key-value pair data located in the new key-value pair partition is flushed. At the same time, the key-value pair partition management module takes over the new key-value pair partition and overwrites the key-value pair data outside the new key-value pair partition and stores it in the key-value pair data temporary storage area.

[0031] The present invention also provides a server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for each step in the above method.

[0032] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0033] The advantages of the present invention are as follows:

[0034] The present invention organizes the sorted string table through dynamic partitioning, determines the key-value pair partition based on cluster analysis, and independently performs merge sorting of the sorted string table on each key-value pair partition, thereby effectively limiting the formation of sorted string tables with sparse metadata distribution, greatly reducing the number of high-level sorted string tables involved in merge sorting, and at the same time, the key-value pair partitioning further accelerates the retrieval and positioning of key-value pair data, thereby improving the read and write performance of the system.

[0035] This method has strong practicality and application scope in the field of Internet data storage and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] FIG1 is a schematic diagram of an LSM data organization method for sparsely distributed metadata provided by the present invention;

[0037] FIG2 is a flowchart of an LSM data flushing process for sparsely distributed metadata provided by the present invention;

[0038] FIG3 is a schematic diagram of an LSM data organization device for sparsely distributed metadata provided by the present invention;

[0039] FIG4 is a comparison chart of throughput performance between the present invention and the prior art under different workload scenarios;

[0040] FIG5( a ) is a comparison diagram of the actual written data volume when writing data of the same volume size between the present invention and the prior art;

[0041] FIG5( b ) is a comparison diagram of the data size in the merging process between the present invention and the prior art. DETAILED DESCRIPTION

[0042] The present invention will be described in further detail below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0043] Figure 1 is a schematic diagram of an LSM data organization method for sparsely distributed metadata proposed by the present invention. As shown in Figure 1, the mutable memory table in the memory is used to receive key-value pair data written at high speed. When the size of the mutable memory table exceeds the threshold, it will be converted into an immutable memory table. There can be multiple immutable memory tables, which are managed using a flush queue.

[0044] When the key-value pair data is flushed to disk, when the mutable memory table is first triggered to be converted to an immutable memory table, the key name clustering analysis function will be used to perform cluster analysis on all key names in the mutable memory table to obtain several key name groups. Each key name group will be pruned to obtain a key-value pair partition, and the key-value pair data in the key-value pair partition will be flushed to form a sorted string table. At the same time, the key-value pair data outside the key-value pair partition will be stored in the key-value pair data temporary storage area. When the key-value pair data is subsequently flushed to disk, the key-value pair data will be flushed to the corresponding key-value pair partition, and the key names of the remaining key-value pair data and the key-value pair data in the key-value pair data temporary storage area will be obtained. Cluster analysis will be performed to detect whether a new key-value pair partition is generated.

[0045] Each key-value pair partition performs a merge sort operation on the sorted string table independently.

[0046] When obtaining key-value pair data based on a key name query, the system will query the mutable memory table, immutable memory table, and key-value pair data temporary storage area in sequence based on the key name. If the target key name does not exist in any of the above areas, the key-value pair partition to which it belongs is determined based on the key name, and the target key name is further queried in the key-value pair partition to obtain the key-value pair data.

[0047] FIG2 is a flowchart of an LSM data flushing process for sparsely distributed metadata provided by the present invention. As shown in FIG2 , the data flushing process of this embodiment includes:

[0048] S101: Triggering the transformation of a mutable memory table into an immutable memory table.

[0049] S102: List key-value pair partitions.

[0050] Specifically, the key-value pair partition management module is requested to obtain all established key-value pair partitions. The key-value pair partition information includes: the left boundary value of the key control range, the right boundary value of the key control range, the expansion coefficient, the key control range size threshold, the number of sorting string tables, etc. All key-value pair partitions are listed in order of the key control range size.

[0051] S103: Determine whether the key-value pair partition is empty; if it is empty, execute S104; if not, execute S109.

[0052] S104: Obtain the full key name in the variable memory table.

[0053] S105: Perform cluster analysis on the batch of key names to obtain several key name groups.

[0054] Specifically, the Levenshtein distance is used to measure the distance between key names, or a distance measurement standard is selected based on business needs, and a clustering analysis algorithm such as K-Means is used to perform cluster analysis on batch key names to obtain key name groups.

[0055] S106: Determine whether the key name group is not empty; if not, execute S107; if empty, execute S108.

[0056] S107: Prune each key name group to form a new key-value pair partition and manage it. When the immutable memory table is flushed to the disk, the key-value pair data is flushed to the corresponding key-value pair partition.

[0057] Specifically, the key name group is pruned based on the key range size threshold R of the key-value pair partition and the existing key-value pair partitions, so that the target key-value pair partition satisfies the following conditions: 1) the absolute difference between the right boundary value and the left boundary value of the key range remains within the key range size threshold R; and 2) the key range does not intersect with the key range of the existing key-value pair partition. After pruning the key name group to obtain the key-value pair partition, the key-value pair data in the key-value pair partition is flushed to form a sorted string table. At the same time, the key-value pair partition information is managed.

[0058] S108: storing the remaining key-value pair data after the flashing into the key-value pair data temporary storage area.

[0059] Specifically, after the above-mentioned key-value pair data located in the key-value pair partition is flushed, if there is any remaining key-value pair data, it will be stored in the key-value pair data temporary storage area. The key-value pair data temporary storage area uses a data structure that supports fast retrieval, such as a hash table, to manage key-value pair data. At the same time, the key-value pair data temporary storage area provides a full key name and full key-value pair data acquisition interface to the outside world. The external module can quickly obtain all key-value pair data and their key names in the key-value pair data temporary storage area through this interface.

[0060] S109: When the immutable memory table is flushed to the disk, the key-value pair data is flushed to the corresponding key-value pair partition.

[0061] Specifically, the key-value pair data whose key names are in the interval [AT, B+T] are filtered and written to the corresponding key-value pair partition to form a sorted string table, wherein the left and right boundary values ​​of the interval are adjusted as appropriate. If the key-value pair partition i where the key name of the currently filtered key-value pair is located has a previous key-value pair partition j, then the left boundary value AT of the interval needs to be greater than the right boundary value of the key control range of the key-value pair partition j. If the key-value pair partition i has a subsequent key-value pair partition k, then the right boundary value B+T of the interval needs to be less than the left boundary value of the key control range of the key-value pair partition k, and the size of the interval B-A+2*T needs to be kept within the key control range size threshold R of the key-value pair partition.

[0062] S110: Update information such as the key control range of the key-value pair partition.

[0063] S111: Acquire the key-value pair data remaining after the above-mentioned flashing and the key names of the key-value pair data in the key-value pair data temporary storage area.

[0064] Specifically, access the full key name acquisition interface provided by the immutable memory table to quickly obtain the key name list of the key-value pair data remaining after the flush, and access the full key name acquisition interface provided by the key-value pair data temporary storage area to quickly obtain the key names of all key-value pair data in the temporary storage area.

[0065] Figure 3 is a schematic diagram of an LSM data organization device for sparsely distributed metadata provided by the present invention. As shown in Figure 3, the device of this embodiment includes: a key name acquisition module, a key name clustering analysis module, a key-value pair partition management module, a key-value pair data temporary storage area module and a key-value pair data flushing module.

[0066] The key name acquisition module is specifically used to obtain the key name of the key-value pair data in the mutable memory table and the immutable memory table, or to obtain the key name of the key-value pair data in the key-value pair data temporary storage area. The module encapsulates the above-mentioned multiple operations for obtaining the key name of the key-value pair data internally and provides a unified key name acquisition interface externally.

[0067] The key name clustering analysis module is specifically used to perform cluster analysis on batch key names and return key-value pair partitions. The module internally includes a module interaction unit, a clustering analysis algorithm unit, a key name group pruning unit, etc. The module interaction unit is responsible for receiving batch key names and feeding back the key-value pair partition results to the calling module; the clustering analysis algorithm unit uses the Levenshtein distance to measure the distance between key names, and uses clustering analysis algorithms such as K-Means to process batch key names to obtain key name groups; the key name group pruning unit further prunes the key name groups based on the key control range size threshold R of the key value pair partition and the existing key value pair partition to obtain the key value pair partition, so that it satisfies in sequence: there is no intersection between the key control range and the key control range of the existing key value pair partition, and the absolute difference between the right boundary value and the left boundary value of the key control range remains within the key control range size threshold R.

[0068] The key-value partition management module is specifically used to store and manage key-value partition information. This information is stored in a data table, with table fields including: ID, key range left boundary value, key range right boundary value, expansion coefficient, key range size threshold, number of sorted string tables, etc. At the same time, key-value partition information is managed by manipulating data table records, including adding new key-value partitions, updating key-value partition information, and listing key-value partitions. The atomicity of these operations must be ensured.

[0069] The key-value data temporary storage area module is specifically used to store key-value data outside the key-value partition during system operation. The module is located in the memory and is mainly composed of a data structure that supports fast retrieval, a list that stores the full amount of key-value data sorted by key name, and a list that stores the full amount of key names. The data structure that supports fast retrieval can be implemented using, for example, a hash table. At the same time, the module is configured with a related logging mechanism, such as a WAL (write-ahead log) mechanism, so as to achieve consistency, persistence and reliability of the key-value data in the module.

[0070] The key-value data flushing module is specifically used to integrate various modules to complete the key-value data flushing. After the module receives the command to convert the variable memory table into the immutable memory table, it initiates a key-value partition list acquisition request to the key-value partition management module, queries the key-value partition data table and returns the full record package. If the key-value partition list is empty, it initiates a key name acquisition request to the key name acquisition module, and the key name acquisition module uniformly obtains the full key name of the key-value data in the variable memory table, and transmits it to the key name clustering analysis module for clustering analysis to obtain the key-value partition, flush the key-value data within the key-value partition, and stores the key-value data outside the key-value partition in the key-value data temporary storage area module. The zone management module manages the key-value pair partitions. If the key-value pair partition list is not empty, when the immutable memory table is flushed to the disk, the key-value pair data in the immutable memory table is flushed to the corresponding key-value pair partition to obtain the remaining key-value pair data M, and again initiates a key name acquisition request to the key name acquisition module to obtain the full key name of the key-value pair data N and the key name of M in the key-value pair data temporary storage area module, and merges and transmits them to the key name clustering analysis module for clustering analysis. If a new key-value pair partition is obtained, M and N are flushed to the corresponding key-value pair partition again, the key-value pair data temporary storage area is reset, and the key-value pair data outside the new key-value pair partition is stored. The key-value pair partition management module manages the new key-value pair partition.

[0071] A practical application scenario of this invention involves two data import services, A and B, using different file naming conventions, writing files to the same data table. To verify the effectiveness of this invention in practice, we implemented it in the open-source LSM software HBase, denoted as dpstore, and conducted performance tests and comparisons with the default HBase (denoted as default).

[0072] Specifically, to minimize the impact of factors like network transmission and node communication in a distributed model on the experimental results, HBase was deployed in a standalone mode. Furthermore, to accurately compare the read and write performance of dpstore and default, all software configuration parameters except the merge strategy were kept consistent.

[0073] Specifically, performance testing was conducted using the benchmark tool YCSB. We simulated scenarios where businesses A and B were importing files using custom key name generation rules. Specifically, business A generated key names based on digits 0-9, while business B generated key names based on the letters a-z. We also compared throughput performance under several workload scenarios: Load (100% write), workloada (50% read, 50% update), workloadb (95% read, 5% update), workloadc (100% read), workloadd (95% read, 5% write), workloade (95% range query, 5% write), and workloadf (50% read, 50% RMW).

[0074] Specifically, the size of each file is 10KB, the number of files loaded is 20 million, and the number of operations is 2 million.

[0075] The server performance parameter information used in the effectiveness verification experiment of the present invention is shown in Table 1:

[0076] Table 1 Server performance parameter information

[0077] The software version information used in the effectiveness verification experiment of the present invention is shown in Table 2:

[0078] Table 2 Software version information

[0079] The experimental test results are shown in Figure 4, which shows that dpstore has better throughput performance than default in every workload scenario, with data loading performance improved by 11%.

[0080] Furthermore, Figure 5(a) shows a comparison of the actual data write volume when dpstore and default write data of the same size in a single pass, that is, a write amplification comparison. The results show that dpstore reduces the actual data write volume compared to default. Specifically, when writing 10GB, 48GB, 95GB, 143GB, and 191GB files, the actual data write volume of default is 38GB, 131GB, 280GB, 523GB, and 801GB, respectively, while the actual data write volume of dpstore is 12GB, 84GB, 225GB, 403GB, and 516GB, respectively. The actual data write volume is reduced by an average of 37%.

[0081] Figure 5(b) shows a comparison of the data size in the dpstore and default merges, with the merge number on the horizontal axis and the size range of the ordered string files in each merge on the vertical axis. It can be seen that the dpstore merge contains smaller files than the default merge, with the majority being 40-50MB files. The default merge contains a file with a maximum size of 2500MB.

[0082] While specific embodiments of the present invention have been disclosed for illustrative purposes, intended to facilitate understanding and implementation of the present invention, those skilled in the art will appreciate that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the disclosure of the preferred embodiments, and the scope of protection claimed in the present invention shall be determined by the scope of the claims.

Claims

1. A LSM data organization method for sparsely distributed metadata, comprising the following steps: Receive written key-value pair data through the mutable memory table in memory; When the variable memory table reaches a threshold, convert it into one or more immutable memory tables; Writing the key-value pair data in the plurality of immutable memory tables to disk; Its characteristics are: When the mutable memory table is first triggered to be converted to an immutable memory table, cluster analysis is performed on the key names of the key-value pair data in the immutable memory table based on the key name distance to obtain a plurality of key name groups; each of the key name groups is pruned to obtain a key-value pair partition; the key-value pair data in the key-value pair partition is flushed to form a sorted string table, and the key-value pair data outside the key-value pair partition is uniformly stored in a key-value pair data temporary storage area in the memory area; When the mutable memory table is converted to an immutable memory table but not triggered for the first time, the key-value pair partitions are listed in order of the key-controlled range size, and information of each key-value pair partition is obtained, including the key-controlled range [A, B] of the key-value pair partition, the expansion coefficient T of the key-value pair partition, and the key-controlled range size threshold R of the key-value pair partition; when the disk is to be flushed, the key-value pair data in the immutable memory table is retrieved, the key-value pair data with the key name in the interval [AT, B+T] is filtered out, and is flushed to the corresponding key-value pair partition to form a sorted string table and the key-controlled range of the key-value pair partition is updated; the key names of the remaining filtered key-value pair data and the key-value pair data in the key-value pair data temporary storage area are obtained, and clustered to obtain a new key name group; a new key-value pair partition is obtained based on the new key name group, and the key-value pair data in the new key-value pair partition is flushed to form a sorted string table.

2. The method according to claim 1, characterized in that The method for pruning each key name group is: taking the minimum key in the key name group as the left boundary value of the key control range of the corresponding key-value pair partition, and taking the maximum key in the key name group as the right boundary value of the key control range of the key-value pair partition; if the absolute difference between the right boundary value and the left boundary value of the key control range of the key-value pair partition exceeds the key control range size threshold R of the key-value pair partition, then the key control range of the key-value pair partition is pruned according to the key control range size threshold R of the key-value pair partition, and the pruning method is to eliminate the minimum key in sequence according to the key size until the absolute difference between the right boundary value and the left boundary value of the key control range of the key-value pair partition remains within the key control range size threshold R of the key-value pair partition.

3. The method according to claim 1 or 2, characterized in that The absolute difference between the right boundary value and the left boundary value of the key control range of the new key-value pair partition remains within the key control range size threshold R of the key-value pair partition, and there is no intersection between the key control ranges of the new key-value pair partition and the existing key-value pair partition.

4. The method according to claim 1, wherein The key-value pair data temporary storage area is implemented using a set retrieval data structure, and the application supports fast retrieval; the key-value pair data temporary storage area provides an interface for external modules to obtain the full amount of key-value pair data and full key names in the temporary storage area.

5. The method according to claim 1, wherein In the process of filtering out key-value pair data with key names in the interval [AT, B+T], if the key-value pair partition i where the key name of the currently filtered key-value pair is located has a previous key-value pair partition j, then the left boundary value AT of the interval needs to be greater than the right boundary value of the key control range of the key-value pair partition j; if the key-value pair partition i has a subsequent key-value pair partition k, then the right boundary value B+T of the interval needs to be less than the left boundary value of the key control range of the key-value pair partition k, and the size of the interval B-A+2*T needs to be kept within the key control range size threshold R of the key-value pair partition.

6. The method according to claim 1, characterized in that When there are multiple immutable memory tables, a flush queue is used to manage the immutable memory tables.

7. An LSM data organization device for sparsely distributed metadata, characterized in that: include Key name acquisition module, used to obtain the key name of key-value pair data; The key name clustering analysis module is used to perform cluster analysis on the key names of the key-value pair data in the immutable memory table according to the key name distance when the mutable memory table is first triggered to be converted to the immutable memory table, and obtain a number of key name groups; prune each key name group to obtain a key-value pair partition; when the mutable memory table is not first triggered to be converted to the immutable memory table, list the key-value pair partitions in order of key control range size, and obtain information about each key-value pair partition, including the key control range [A, B] of the key-value pair partition, the expansion coefficient of the key-value pair partition, and the key control range of the key-value pair partition. The number T and the key control range size threshold R of the key-value pair partition are selected; when writing to the disk, the key-value pair data in the immutable memory table are retrieved, the key-value pair data with the key name in the interval [AT, B+T] are filtered out, and are written to the corresponding key-value pair partition to form a sorted string table and update the key control range of the key-value pair partition; the key names of the key-value pair data remaining after the screening and the key-value pair data in the key-value pair data temporary storage area are obtained, and clustered to obtain a new key name group again; then, a new key-value pair partition is obtained based on the new key name group; A key-value pair partition management module is used to store and manage key-value pair partition information. If the key-value pair partition list is not empty, the key-value pair data is flushed to the corresponding key-value pair partition to form a sorted string table. The remaining key-value pair data and the key names of the key-value pair data in the key-value pair data temporary storage area module are transmitted to the key-value pair clustering analysis module, and the key-value pair data in the new key-value pair partition is flushed. The key-value pair partition information includes a key control range, an amplification coefficient, a key control range size threshold, and the number of sorted string tables. A key-value pair data temporary storage area module, used for storing key-value pair data outside the key-value pair partition; The key-value pair data flushing module is used to coordinate the key-value pair data flushing process. First, it obtains the key-value pair partition list from the key-value pair partition management module. If the key-value pair partition list is empty, it obtains the key names of all key-value pair data in the variable memory table from the key name acquisition module and transmits them to the key-value pair clustering analysis module; the key-value pair data located outside the key-value pair partition is stored in the key-value pair data temporary storage area module.

8. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images