L0 layer data processing method and device of LSM-tree storage

Abstract

The present application relates to the field of storage, and provides an L0 layer data processing method and device for LSM-tree storage, which reserves a part of an area in persistent memory as an L0 layer according to an original capacity of the L0 layer, and the method comprises the following steps: determining relevant statistical information of a load in memory, wherein the relevant statistical information of the load comprises the number of times of writing the load into each key value in the L0 layer; performing vertical partitioning on L0 layer data according to the relevant statistical information of the load and a concurrent degree set by a user, and writing the starting physical address of each partition into a metadata file; and writing data corresponding to the load into the L0 layer according to the partitioning result and the metadata file, and returning to the above steps of determining the relevant statistical information of the load to be written into the L0 layer in the memory and the subsequent steps. The present application can improve the query and merging efficiency of L0 layer data in an SM-tree structure.

Description

Technical Field

[0001] This article relates to the field of data storage, and in particular to a method and apparatus for L0 layer data processing based on persistent memory LSM-tree storage. Background Technology

[0002] LSM-tree is a data storage structure used in HDDs / SSDs. It ensures high read performance by converting random writes to sequential writes, making it the primary data structure in some database storage engines, replacing B+Tree. However, the multi-level structure of LSM-trees sacrifices some read performance and increases the unique merging overhead. Merging operations in LSM-trees can affect the response time of some read operations and consume CPU resources, making it crucial to shorten merging time. While LSM-trees reduce the amount of data accessed by ensuring that the key-value ranges of SSTables in levels after L1 do not overlap, considering the smaller amount of data in the L0 layer, the lack of byte addressing support in HDDs / SSDs, and the poor performance of random writes, the data in the L0 layer is not partitioned, resulting in overlapping SSTable key-value ranges. Therefore, during data querying and merging, the parallel capabilities of threads cannot be utilized to improve query and merging efficiency. Summary of the Invention

[0003] This paper addresses the problem of low query and merging efficiency of L0 layer data in the LSM-tree structure of HDD / SSD in existing technologies.

[0004] To address the aforementioned technical problems, this paper firstly provides a method for processing L0 layer data in LSM-tree storage, which reserves a portion of the original L0 layer capacity in persistent memory as the L0 layer. The method includes:

[0005] Determine relevant statistics about the load in memory, wherein the relevant statistics about the load include the number of times the load writes to each key value;

[0006] Based on the relevant statistics of the load and the concurrency level set by the user, the L0 layer data is vertically partitioned, and the starting physical address of each partition is written into the metadata file.

[0007] Based on the partitioning results and metadata files, the load-related data is written to the L0 layer, and the process of determining the load-related statistics in memory and subsequent steps is repeated.

[0008] As a further embodiment of this paper, the relevant statistical information for determining the load to be written to the L0 layer in memory includes:

[0009] Receive relevant statistical information about the load to be written to the L0 layer in memory from the front end; or use the count-min algorithm to estimate relevant statistical information about the load to be written to the L0 layer in memory.

[0010] As a further embodiment of this paper, the relevant statistical information of the load to be written to L0 layer in memory using the count-min algorithm includes:

[0011] Get d hash functions with identical value ranges specified by the user, and initialize a d-row, w-column array C[d][w], where w is the number of distinct elements in the value range of the hash function;

[0012] For each key-value j in memory, iterate through all d hash functions and execute C[i][hi(j)], where h i (j) represents the value of key j in the i-th hash function;

[0013] For any key value 'a', the h-th element of array C is... i The minimum value in column (a) is the number of times the load is written to the key value a in L0.

[0014] In a further embodiment of this paper, the vertical partitioning of the L0 layer data based on relevant load statistics and user-defined concurrency levels includes:

[0015] Calculate the total amount of key values ​​written by the load based on relevant load statistics;

[0016] Sort the key values ​​written by the load in ascending order;

[0017] Based on the user-defined concurrency level, the total number of key values ​​written to the load, and the sorting results, determine the partition keys and write the physical addresses of the partition keys to the metadata file; determine the partitioning results based on the partition keys and the sorting results.

[0018] In a further embodiment of this paper, the partition keys are determined based on the user-defined concurrency level, the total number of key values ​​written to the load, and the sorting results.

[0019] The average data volume of a partition is calculated by dividing the total number of key values ​​written by the load by the user-defined concurrency level.

[0020] The key value in the sorting result that satisfies the following formula is used as the partition key: n×X+1, where n is a positive integer from 1 to N-1, N is the concurrency level, and X is the average data size of the partition.

[0021] In a further embodiment of this paper, after determining the partitioning result based on each partition key and the sorting result, the method further includes:

[0022] Determine if there are identical key values ​​in adjacent partitions. If so, assign the key value to the partition with the smaller data volume or the partition with more identical key values ​​in the adjacent partition.

[0023] As a further embodiment of this paper, after vertically partitioning the L0 layer data according to relevant load statistics and user-defined concurrency levels, the method further includes:

[0024] Monitor whether the difference in data volume between each partition exceeds a predetermined threshold. If so, use the count-min algorithm to re-estimate the relevant statistics of the load to be written to the L0 layer in memory.

[0025] As a further embodiment of this paper, the processing of L0 layer data stored in the LSM-tree also includes:

[0026] Receive a query operation request, wherein the query operation request includes at least a request key value;

[0027] The request key value is queried from the metadata file to determine the partition to which the request key value falls;

[0028] Determine the key-value information under the partition based on the partition it falls into;

[0029] Based on the requested key value, a binary search is used to perform a query operation on the key value information in this partition.

[0030] In a further embodiment of this paper, reserving a portion of persistent memory as the L0 layer based on the original capacity of the L0 layer includes:

[0031] The partial area is calculated using the following formula: X = Y + a, where X is the partial area, Y is the original capacity of layer L0, and a is a value greater than 0.

[0032] The space of the aforementioned region is reserved in persistent memory as the L0 layer.

[0033] As a further embodiment of this paper, the processing of L0 layer data stored in the LSM-tree also includes:

[0034] When the L0 layer data is full, the merging operation is performed according to the following procedure:

[0035] Read the data from L0 and L1 layers into memory;

[0036] Create a number of threads equal to the number of partitions in L0. Each thread selects the overlapping parts in L1 layer for each L0 partition in memory, merges the data in this part, writes the merged data to L1 layer, and updates the metadata file.

[0037] The second aspect of this paper provides an L0 layer data processing device for LSM-tree storage, which reserves a portion of the original L0 layer capacity in persistent memory as the L0 layer, including:

[0038] A statistics unit is used to determine relevant statistics about the load in memory, wherein the relevant statistics about the load include the number of times the load writes to each key value;

[0039] The partitioning unit is used to vertically partition the L0 layer data based on relevant load statistics and user-defined concurrency levels, and write the starting physical address of each partition into the metadata file.

[0040] The write response unit is used to write the load-related data into the L0 layer based on the partitioning results and metadata file, and then return to repeat the above steps of determining the load-related statistics in memory and subsequent steps.

[0041] A third aspect of this document provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in any of the foregoing embodiments.

[0042] A fourth aspect of this document provides a computer storage medium having a computer program stored thereon, which, when executed by a processor of a computer device, performs instructions according to the method described in any of the foregoing examples.

[0043] The fifth aspect of this document provides a computer program product comprising a computer program that, when executed by a processor, implements the method described in any of the foregoing embodiments.

[0044] This paper utilizes a portion of persistent memory to replace the L0 layer in an LSM-tree. By leveraging the performance advantages and addressing capabilities of persistent memory, vertical partitioning of the L0 layer data can be achieved (where key values ​​within a partition overlap, but the intervals do not, or adjacent intervals have overlapping key values). This ensures that the data in each interval is evenly distributed. Partitioning is the smallest granularity for merging and querying operations, and different intervals can be operated on in parallel by different threads. This improves the query efficiency of the LSM-tree at the L0 layer and shortens the time required to merge from L0 to L1.

[0045] To make the above and other objects, features and advantages of this document more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments or prior art described herein, the accompanying drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this article. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 This paper presents a flowchart of a method for processing L0 layer data stored in an LSM-tree according to an embodiment of the present paper.

[0048] Figure 2 This document illustrates a flowchart of an embodiment that uses the count-min algorithm to estimate relevant statistics about the load in memory.

[0049] Figure 3 A flowchart of the vertical partitioning process in the embodiments of this article is shown;

[0050] Figure 4 Another flowchart of the L0 layer data processing method of LSM-tree storage in the embodiments of this paper is shown;

[0051] Figure 5 A flowchart illustrating the merging operation execution process in the embodiments of this article is shown;

[0052] Figure 6 This paper shows a structural diagram of the L0 layer data processing device for LSM-tree storage in an embodiment of the invention;

[0053] Figure 7 A structural diagram of the computer device described in this embodiment is shown.

[0054] Explanation of symbols in the attached drawings:

[0055] 601. Statistical Unit;

[0056] 602. Partition Unit;

[0057] 603. Write to the response unit;

[0058] 702. Computer equipment;

[0059] 704, Processor;

[0060] 706. Memory;

[0061] 708. Drive mechanism;

[0062] 710. Input / Output Module;

[0063] 712. Input devices;

[0064] 714. Output devices;

[0065] 716. Presentation equipment;

[0066] 718. Graphical User Interface;

[0067] 720. Network interface;

[0068] 722. Communication link;

[0069] 724. Communication bus. Detailed Implementation

[0070] The technical solutions in the embodiments described below will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments described herein, and not all of the embodiments. Based on the embodiments described herein, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this document.

[0071] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings herein are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0072] This specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or device products, the methods shown in the embodiments or drawings can be executed sequentially or in parallel.

[0073] It should be noted that the L0 layer data processing method and device of LSM-tree storage in this paper can be used for data storage in the financial field, as well as in any field other than finance. The application field of the L0 layer data processing method and device of LSM-tree storage in this paper is not limited.

[0074] First, let's explain the relevant technical terms used in this article:

[0075] LSM-tree storage: LSM-tree is a commonly used structure in key-value database storage engines. It typically divides data into multiple levels (L0, L1, L2, etc.), with capacity increasing from small to large and time from recent to oldest. Data in higher levels is moved to the next level through compaction. Its design aims to ensure that write operations are appended to disk, avoiding random writes and improving write performance. However, the multi-level structure sacrifices some read performance. Data in each level of an LSM-tree is stored on an HDD / SSD. Level L0 stores data evicted from memory, and the SSTable key-value ranges in L1 and subsequent levels do not overlap.

[0076] L0 layer: The top layer in the LSM-tree, refreshed directly from memory. L0 layer is characterized by the smallest data volume and the most up-to-date data. Unlike other layers, the traditional L0 layer does not perform grouping operations; the SSTable contains all key-value ranges.

[0077] SSTable: In an LSM-tree, data is stored in secondary storage as SSTable files. Each level consists of a varying number of SSTables. SSTables are immutable and consist of multiple ordered data blocks and metadata.

[0078] Merge: Read all SSTables from the upper layer (L0 is the upper layer of L1) and the overlapping SSTables from the lower layer (L2 is the lower layer of L1) into memory, sort them to ensure they are ordered, write them as new SSTables to the lower layer, and delete the SSTables from the upper and lower layers that participated in the merge.

[0079] Existing LSM-trees suffer from overlapping SSTable key-value ranges in their L0 layer, hindering the use of thread parallelism to improve query and merging efficiency. To address these issues, this paper provides a method for processing L0 layer data in an LSM-tree storage system. This method pre-reserves a portion of the original L0 layer capacity in persistent memory as the L0 layer. Figure 1 As shown, the L0 layer data processing method stored in the LSM-tree includes:

[0080] Step 101: Determine the relevant statistical information of the load in memory, wherein the relevant statistical information of the load includes the number of times the load writes to each key value;

[0081] Step 102: Based on the relevant statistical information of the load and the concurrency level set by the user, the L0 layer data is vertically partitioned, and the starting physical address of each partition is written into the metadata file.

[0082] Step 103: Based on the partitioning results and metadata file, write the load-related data into the L0 layer, return to step 101, and repeat the above steps of determining the load-related statistical information to be written into the L0 layer in memory and the subsequent steps.

[0083] In detail, the methods described herein are applicable to terminal devices with storage devices, including but not limited to desktop computers, laptops, servers, etc.

[0084] The load described in this article can be set by business requirements, such as the inventory of a certain software or company, or it can be determined manually. This article does not impose specific limitations on the load. The key values ​​written to the load represent the specific load type. For example, key value 1 represents the inventory of a certain product, which can be pre-configured according to requirements.

[0085] When implementing step 101, the front end can determine relevant statistics about the load in memory through statistical analysis, and the count-min algorithm can also be used to estimate relevant statistics about the load in memory. Specifically, such as... Figure 2 As shown, the relevant statistics for estimating memory load using the count-min algorithm include:

[0086] Step 201: Obtain d hash functions with identical value ranges specified by the user, and initialize a d-row, w-column array C[d][w], where w is the number of distinct elements in the value range of the hash function;

[0087] Step 202: For each key-value j in memory, iterate through all hash functions and execute C[i][hi(j)], where h i (j) represents the value of key j in the i-th hash function;

[0088] Step 203: For the occurrence count of any key value 'a', the h-th element of array C is... i The minimum value in column (a) is the number of times the load is written to the key value a in L0.

[0089] In terms of implementation, considering the possibility of writing the same key value multiple times and the inaccuracy of the count-min algorithm, a space slightly larger than the original L0 layer can be reserved in persistent memory as a new L0 layer.

[0090] The vertical partitioning in step 102 results in overlapping key values ​​within partitions, while the intervals themselves do not overlap, or adjacent intervals may have overlapping key values. Since intervals do not overlap or adjacent intervals overlap, during data querying, it is only necessary to first determine the partition containing the relevant key value and then query the key value again from the relevant partition, thus avoiding a complete retrieval of the L0 layer and improving data query efficiency. Furthermore, when merging L0 and L1 layer data, multiple threads can be started for each partition, with each thread performing the merge operation on each partition, thereby improving data merging efficiency.

[0091] The starting physical address of each partition refers to the physical address of the first key value in each partition. In practice, in addition to the starting physical address of each partition, the metadata file also records the maximum key value and offset in the data block of each partition. During data access, a binary search can be performed using the metadata file to locate the relevant data. The binary search divides the ordered data into two equal parts (front and back) from the middle. The data to be searched is compared with the data in the middle. If they are equal, the search is successful; if the data is smaller than the middle data, the search continues recursively to the first half of the data; otherwise, it continues to the second half (assuming the data is in ascending order) until the search is successful.

[0092] When implementing step 103, according to the partitioning order, the corresponding <key value, data> is written to layer L0 in turn based on the key value in the partitioning result.

[0093] For example, suppose the relevant statistics of a certain load are: the load writes key value 1 and key value 6 twice, the load writes key value 3, key value 5, key value 9, and key value 12 once each, and the user-specified concurrency level is 4. Then it can be determined that the L0 layer data can be divided into four partitions: {1,1}, {3,5}, {6,6}, and {9,12}. The key values ​​in this embodiment are only examples. In specific implementation, other specific values ​​can be set.

[0094] This embodiment utilizes a portion of persistent memory to replace the L0 layer in the LSM-tree. Leveraging the performance advantages and addressing capabilities of persistent memory, it achieves vertical partitioning of the L0 layer data while ensuring a uniform distribution of data across each partition. Partitioning is the smallest granularity for merging and querying operations, and different partitions can be operated on in parallel by different threads. This improves the query efficiency of the LSM-tree at the L0 layer and shortens the time required to merge from L0 to L1. Furthermore, by repeatedly determining the partitioning results, this paper achieves dynamic maintenance of the partitioning results, ensuring the balance of the partitioning results and thus guaranteeing the efficiency of data merging and querying.

[0095] In one embodiment of this article, such as Figure 3 As shown, step 102 above, which involves vertically partitioning the L0 layer data based on relevant load statistics and user-defined concurrency levels, includes:

[0096] Step 301: Calculate the total amount of key values ​​written by the load based on the relevant statistical information of the load;

[0097] Step 302: Sort the key values ​​of the load write in ascending order;

[0098] Step 303: Based on the user-defined concurrency level, total number of key values ​​written to the load, and sorting results, determine each partition key and write the physical address of the partition key to the metadata file; determine the partitioning results based on each partition key and sorting results.

[0099] When implementing step 301, the total number of key values ​​written by the load is calculated by summing the number of times each key value is written in the relevant statistics of the load. For example, if the relevant statistics of the load are: the load writes key value 1 and key value 6 twice, and the load writes key value 3, key value 5, key value 9 and key value 12 once each, then the total number of key values ​​written by the load is 2+2+1+1+1+1+1=8.

[0100] When implementing step 302, taking the previous statistical information as an example, the sorting result is key value 1, key value 1, key value 3, key value 5, key value 6, key value 6, key value 9, key value 12.

[0101] In step 303, the average data volume of a partition is calculated by dividing the total number of key values ​​written by the load by the user-defined concurrency level. Then, the key values ​​in the sorting results that satisfy the following formula are used as partition keys: n×X+1, where n is a positive integer from 1 to N-1, N is the concurrency level, and X is the average data volume of the partition. Continuing with the example from the previous step, the partition keys are: key value 3, key value 6, and key value 9.

[0102] When calculating the average data volume, if there are cases where the division is not exact, rounding up or down is performed. For example, if the user sets the concurrency level to 4, and the relevant load statistics are: the load writes key value 1 twice, key value 6 three times, and key values ​​3, 5, 9, and 12 once each, the sorting result is key value 1, key value 1, key value 3, key value 5, key value 6, key value 6, key value 6, key value 9, key value 12. The total number of key values ​​written is 9. Rounding down 9 / 4 gives 2, so the partition keys are key values ​​3, 6, 6, and 12, and the partitioning results are: {1,1}, {3,5}, {6,6}, {6,9}, {12}. In one embodiment of this paper, to avoid adjacent partition keys having the same key value, which would affect data query efficiency, after determining the partitioning results based on each partition key and the sorting results, the key values ​​within the partitions are fine-tuned, specifically including:

[0103] The algorithm checks if there are identical key values ​​in adjacent partitions. If so, it assigns the key value to the partition with the smaller data volume or the partition with more identical key values. Continuing with the previous example, the key value 6 in {6,9} will be assigned to the previous partition {6,6}, resulting in the partitions {1,1}, {3,5}, {6,6,6}, {9}, and {12}.

[0104] In one embodiment of this paper, fine-tuning the key values ​​within a partition can lead to significant differences between partitions, thus affecting merging and query efficiency. To address this issue, step 102, after vertically partitioning the L0 layer data based on relevant load statistics and user-defined concurrency levels, further includes:

[0105] Monitor whether the difference in data volume between each partition exceeds a predetermined threshold. If so, use the count-min algorithm to re-estimate the relevant statistics of the load to be written to the L0 layer in memory.

[0106] In one embodiment of this document, reserving a portion of persistent memory as the L0 layer based on the original capacity of the L0 layer includes:

[0107] The partial region is calculated using the following formula: X = Y + a, where X is the partial region, Y is the original capacity of the L0 layer, and a is a value greater than 0, which can be set according to the actual situation; the space of the partial region is reserved in persistent memory as the L0 layer.

[0108] This embodiment avoids the problem of data not being able to be stored due to inaccurate calculations in the count-min algorithm by reserving a space in persistent memory that is slightly larger than the original L0 region as the updated L0.

[0109] In one embodiment of this article, such as Figure 4 As shown, the L0 layer data processing method for LSM-tree storage also includes:

[0110] Step 401: Receive a query operation request, wherein the query operation request includes at least a request key value;

[0111] Step 402: Query the request key value from the metadata file to determine the partition where the request key value falls;

[0112] Step 403: Determine the key-value information under the partition based on the partition into which the data was entered;

[0113] Step 404: Based on the requested key value, perform a binary search to query the key value information in this partition.

[0114] When step 401 is performed, the query operation request is initiated by the user.

[0115] When implementing step 402, assuming the partitioning results are {1,1}, {3,5}, {6,6}, and {9,12}, then the key value 4 is requested to fall in the second partition {3,5}, and the key value 12 is requested to fall in the fourth partition {9,12}.

[0116] Considering that the partition key still contains overlapping key values ​​from multiple writes, a key-value version number can be set. That is, each time data is written, in addition to recording (key value, data), (key value, version number) is also recorded. Concurrent binary search is performed based on (key value, version number), prioritizing the comparison of key values. For example, if (key value, version number) are (3,10), (4,1), and (4,10), then the search is performed in the following order: (3,10) < (4,1) < (4,10).

[0117] This paper makes it possible for query operations (read operations) to no longer need to access all data in the L0 layer, while the addressing capability of persistent memory also allows binary search to be performed at a finer granularity (such as tuple granularity).

[0118] In one embodiment of this paper, when the L0 layer data is full, a merging operation between L0 and L1 also needs to be performed. Specifically, as follows: Figure 5 As shown, the merge operation includes:

[0119] Step 501: Read the data from L0 and L1 layers into memory;

[0120] Step 502: Create a number of threads equal to the number of partitions in L0. Each thread selects the overlapping part in L1 layer for each L0 partition in memory, merges this part of the data, writes the merged data into L1 layer and updates the metadata file.

[0121] The design of this embodiment can greatly improve the parallelism of the merge operation and effectively shorten the time required for merging.

[0122] In addition, it should be noted that because the partitions of L0 and L1 layers may not completely match, there may be multiple L0 layers that require partially overlapping L1 layer partitions when merging. In this case, it is simply necessary to put these multiple L0 layer partitions together and perform the merge operation. In extreme cases, where the partitions required by all L0 layer partitions overlap, the operation degenerates into the original merge operation.

[0123] Based on the same inventive concept, this document also provides an L0 layer data processing device for LSM-tree storage, as described in the following embodiments. Since the principle of the L0 layer data processing device for LSM-tree storage is similar to that of the L0 layer data processing method for LSM-tree storage, the implementation of the L0 layer data processing device for LSM-tree storage can refer to the L0 layer data processing method for LSM-tree storage; repeated details will not be elaborated further.

[0124] Specifically, before implementing this embodiment, a portion of the persistent memory is reserved as the L0 layer based on its original capacity, such as... Figure 6 As shown, the L0 layer data processing device stored in the LSM-tree includes:

[0125] The statistics unit 601 is used to determine relevant statistics about the load in memory, wherein the relevant statistics about the load include the number of times the load is written to each key value in the L0 layer;

[0126] Partitioning unit 602 is used to vertically partition the L0 layer data according to the relevant statistical information of the load and the concurrency level set by the user, and write the starting physical address of each partition into the metadata file.

[0127] The write response unit 603 is used to write the load-related data into the L0 layer according to the partitioning results and metadata file, and return the above steps of determining the load-related statistical information to be written into the L0 layer in memory and the subsequent steps.

[0128] This embodiment utilizes a portion of persistent memory to replace the L0 layer in the LSM-tree. Leveraging the performance advantages and addressing capabilities of persistent memory, it enables vertical partitioning of the L0 layer data (where key values ​​overlap within partitions, but do not overlap between partitions, or where key values ​​overlap between adjacent partitions). Simultaneously, it ensures a uniform distribution of data across each partition. Partitioning is the smallest granularity for merging and querying operations, and different partitions can be operated on in parallel by different threads. This improves the query efficiency of the LSM-tree at the L0 layer and shortens the time required to merge from L0 to L1 layers.

[0129] In a further embodiment, the L0 layer data processing device of the LSM-tree storage further includes: a query response unit, configured to receive a query operation request, wherein the query operation request includes at least a request key value; query the request key value from the metadata file to determine the partition in which the request key value falls; determine the key value information under the partition in which the request key value falls; and perform a query operation on the key value information under the partition in which the request key value falls using a binary search method.

[0130] In a further embodiment, the L0 layer data processing device of the LSM-tree storage further includes: a merge response unit, used to perform a merge operation according to the following process when the L0 layer data is full: read the L0 layer and L1 layer data into memory; create threads equal to the number of L0 partitions, each thread selects the overlapping part of the L1 layer for each L0 partition in memory, and performs merge processing on this part of data, writes the merged data into the L1 layer and updates the metadata file.

[0131] In summary, this paper redesigns the data placement strategy for the L0 layer of an LSM-tree by leveraging persistent memory features, achieving the following two objectives:

[0132] 1) By partitioning, the amount of data that needs to be accessed in each query is reduced, thereby improving the query efficiency of Lsm-tree at the L0 level.

[0133] 2) By performing parallel merging operations in intervals, the time required for merging from L0 to L1 layers is shortened.

[0134] In one embodiment of this document, a computer device is also provided, such as... Figure 7As shown, computer device 702 may include one or more processors 704, such as one or more central processing units (CPUs), each of which may implement one or more hardware threads. Computer device 702 may also include any memory 706 for storing information of any kind, such as code, settings, data, etc. Non-limitingly, for example, memory 706 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Furthermore, any memory may provide volatile or non-volatile retention of information. Furthermore, any memory may represent a fixed or removable component of computer device 702. In one case, when processor 704 executes associated instructions stored in any memory or combination of memories, computer device 702 may perform any operation of the associated instructions. Computer device 702 also includes one or more drive mechanisms 708 for interacting with any memory, such as hard disk drive mechanisms, optical disk drive mechanisms, etc.

[0135] Computer device 702 may also include an input / output module 710 (I / O) for receiving various inputs (via input device 712) and providing various outputs (via output device 714). A specific output mechanism may include a presentation device 716 and an associated graphical user interface 718 (GUI). In other embodiments, the input / output module 710 (I / O), input device 712, and output device 714 may be omitted, and the device may function solely as a computer device within a network. Computer device 702 may also include one or more network interfaces 720 for exchanging data with other devices via one or more communication links 722. One or more communication buses 724 couple the components described above together.

[0136] Communication link 722 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 722 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0137] Corresponding to Figures 1-5 In addition to the methods described above, this embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described methods.

[0138] This embodiment also provides a computer-readable instruction, wherein when a processor executes the instruction, the program therein causes the processor to perform the following: Figures 1 to 5 The method shown.

[0139] It should be understood that in the various embodiments of this document, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this document.

[0140] It should also be understood that, in the embodiments herein, the term "and / or" is merely a description of the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following associated objects have an "or" relationship.

[0141] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this document.

[0142] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0143] In the embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, or they may be electrical, mechanical, or other forms of connection.

[0144] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments described herein, depending on actual needs.

[0145] Furthermore, the functional units in the various embodiments of this document can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0146] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this paper, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this paper. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] This document uses specific embodiments to illustrate the principles and implementation methods of this document. The descriptions of the embodiments above are only for the purpose of helping to understand the methods and core ideas of this document. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this document. Therefore, the content of this specification should not be construed as a limitation of this document.

Claims

1. A method for processing L0 layer data stored in an LSM-tree, characterized in that, The method involves reserving a portion of the original capacity of the L0 layer in persistent memory as the L0 layer, and the method includes: Determine relevant statistics about the load in memory, wherein the relevant statistics about the load include the number of times the load writes to each key value; Based on the relevant statistics of the load and the concurrency level set by the user, the L0 layer data is vertically partitioned, and the starting physical address of each partition is written into the metadata file. Based on the partitioning results and metadata files, the load-related data is written to the L0 layer, and the process of determining the load-related statistics in memory and subsequent steps is repeated. Specifically, based on relevant load statistics and user-defined concurrency levels, the L0 layer data is vertically partitioned, including: Calculate the total amount of key values ​​written by the load based on relevant load statistics; Sort the key values ​​written by the load in ascending order; Based on the user-defined concurrency level, the total number of key values ​​written to the load, and the sorting results, determine the partition keys and write the physical addresses of the partition keys to the metadata file; determine the partitioning results based on the partition keys and the sorting results. Based on the user-defined concurrency level, the total number of key values ​​written to the load, and the sorting results, the partition keys are determined as follows: The average data volume of a partition is calculated by dividing the total number of key values ​​written by the load by the user-defined concurrency level. The key value in the sorting result that satisfies the following formula is used as the partition key: n×X+1, where n is a positive integer from 1 to N-1, N is the concurrency level, and X is the average data size of the partition; After determining the partitioning results based on each partition key and the sorting results, the process also includes: Determine whether there are identical key values ​​in adjacent partitions. If so, assign the key value to the partition with the smaller data volume or the partition with more identical key values ​​in the adjacent partition. This process, which involves vertically partitioning the L0 layer data based on relevant load statistics and user-defined concurrency levels, also includes: Monitor whether the difference in data volume between each partition exceeds a predetermined threshold. If so, use the count-min algorithm to re-estimate the relevant statistics of the load to be written to the L0 layer in memory.

2. The method as described in claim 1, characterized in that, The relevant statistics for determining the load to be written to L0 layer in memory include: Receive relevant statistical information about the load in memory sent by the front end; or estimate relevant statistical information about the load in memory using the count-min algorithm.

3. The method as described in claim 2, characterized in that, The relevant statistics for estimating memory load using the count-min algorithm include: Get d hash functions with identical value ranges specified by the user, and initialize a d-row, w-column array C[d][w], where w is the number of distinct elements in the value range of the hash function; For each key-value j in memory, iterate through all d hash functions and execute C[i][hi(j)], where h i (j) represents the value of key j in the i-th hash function; For any key value 'a', the h-th element of array C is... i The minimum value in column (a) is the number of times the load is written to the key value a in L0.

4. The method as described in claim 1, characterized in that, Also includes: Receive a query operation request, wherein the query operation request includes at least a request key value; The request key value is queried from the metadata file to determine the partition to which the request key value falls; Determine the key-value information under the partition based on the partition it falls into; Based on the requested key value, a binary search is used to perform a query operation on the key value information in this partition.

5. The method as described in claim 1, characterized in that, Based on the original capacity of the L0 layer, a portion of the persistent memory is reserved as the L0 layer, including: The partial area is calculated using the following formula: X = Y + a, where X is the partial area, Y is the original capacity of layer L0, and a is a value greater than 0. The space of the aforementioned region is reserved in persistent memory as the L0 layer.

6. The method as described in claim 1, characterized in that, Also includes: When the L0 layer data is full, the merging operation is performed according to the following procedure: Read the data from L0 and L1 layers into memory; Create a number of threads equal to the number of partitions in L0. Each thread selects the overlapping parts in L1 layer for each L0 partition in memory, merges the data in this part, writes the merged data to L1 layer, and updates the metadata file.

7. A Layer 0 data processing device for LSM-tree storage, characterized in that, Based on the original capacity of the L0 layer, a portion of the persistent memory is reserved as the L0 layer, including: A statistics unit is used to determine relevant statistics about the load in memory, wherein the relevant statistics about the load include the number of times the load writes to each key value; The partitioning unit is used to vertically partition the L0 layer data based on relevant load statistics and user-defined concurrency levels, and write the starting physical address of each partition into the metadata file. The write response unit is used to write the load-related data into the L0 layer according to the partitioning results and metadata file, and return to repeat the above steps of determining the load-related statistics in memory and subsequent steps. Specifically, based on relevant load statistics and user-defined concurrency levels, the L0 layer data is vertically partitioned, including: Calculate the total amount of key values ​​written by the load based on relevant load statistics; Sort the key values ​​written by the load in ascending order; Based on the user-defined concurrency level, the total number of key values ​​written to the load, and the sorting results, determine the partition keys and write the physical addresses of the partition keys to the metadata file; determine the partitioning results based on the partition keys and the sorting results. Based on the user-defined concurrency level, the total number of key values ​​written to the load, and the sorting results, the partition keys are determined as follows: The average data volume of a partition is calculated by dividing the total number of key values ​​written by the load by the user-defined concurrency level. The key value in the sorting result that satisfies the following formula is used as the partition key: n×X+1, where n is a positive integer from 1 to N-1, N is the concurrency level, and X is the average data size of the partition; After determining the partitioning results based on each partition key and the sorting results, the process also includes: Determine whether there are identical key values ​​in adjacent partitions. If so, assign the key value to the partition with the smaller data volume or the partition with more identical key values ​​in the adjacent partition. This process, which involves vertically partitioning the L0 layer data based on relevant load statistics and user-defined concurrency levels, also includes: Monitor whether the difference in data volume between each partition exceeds a predetermined threshold. If so, use the count-min algorithm to re-estimate the relevant statistics of the load to be written to the L0 layer in memory.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 6.

9. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images