An adaptive tuning method and system for LSM-Tree key-value indexes

Abstract

This invention provides an adaptive tuning method and system for LSM-Tree key-value indexes. The method includes: configuring a load-aware module on the index receiving side; acquiring load data through the load-aware module and calculating the write load intensity in the load data; calculating and updating the maximum lazy layer based on the write load intensity; adjusting the storage index balancing strategy and tiering strategy adapted to the write load intensity using the maximum lazy layer based on the actual write load intensity; configuring multiple Bloom filter units for shards, generating Bloom filter unit groups, calculating the bit length index of the Bloom filter unit groups for shards, and controlling the number of Bloom filter unit functions based on the adaptation relationship between the bit length index of the Bloom filter unit groups and the actual write load; and embedding the index into the storage index for key-value access based on the adapted storage index balancing strategy and tiering strategy, and the adapted Bloom filter units.

Description

Technical Field

[0001] This invention relates to the field of query and storage technology for data deduplication, and in particular to an adaptive tuning method and system for LSM-Tree key-value indexes. Background Technology

[0002] With the development of the internet, massive data volumes have brought enormous challenges to data querying and storage. Especially with such large datasets, data deduplication technology has become increasingly important due to its ability to save storage resources. Data deduplication typically performs coarse-grained duplicate detection and eliminates redundant content at the storage level to save storage space. Data deduplication technology is widely used in various storage indexes or in the underlying storage modules of storage management indexes. However, existing data deduplication technologies still fall short of satisfactory data throughput performance, particularly in terms of performance allocation for single-point queries and data writes, where there is room for optimization. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an adaptive tuning method and system for LSM-Tree key-value indexes.

[0004] One objective of this invention is to provide an adaptive tuning method for LSM-Tree key-value indexes, the method comprising:

[0005] Configure a load awareness module at the index data receiving end, obtain load data through the load awareness module, and calculate the write load intensity in the load data;

[0006] The maximum lazy layer is calculated and updated based on the write load intensity. Based on the calculated write load intensity, the expected merging strategy of each key-value index layer adapted to the write load intensity is adjusted using the maximum lazy layer.

[0007] Configure multiple Bloom filter units for fragmentation, generate Bloom filter unit groups, calculate the bit length index of the Bloom filter unit groups for fragmentation, and control the number of Bloom filter units enabled based on the bit length index of the Bloom filter unit groups and the adaptation relationship of the calculated write load.

[0008] Based on the expected merging strategy adapted to the write load intensity, the index status is gradually adjusted during index entry queries and writes to achieve dynamic optimization of the index status.

[0009] Based on the calculated number of Bloom filter units enabled, the number of Bloom filter units loaded into the index memory is dynamically adjusted during index entry querying and writing, thereby dynamically adjusting the total index memory space occupied by the Bloom filter group and achieving dynamic optimization of memory efficiency.

[0010] According to a preferred embodiment of the present invention, the load sensing module is configured at the data receiving end of the LSM-Tree key-value index, and the load sensing module is provided with a load sensing queue, which is used to obtain load data on the input side of the LSM-Tree key-value index, wherein the load data includes write load data and read load data.

[0011] Specifically, a time window of a preset size is configured for the load sensing module, and a load sensing queue is constructed. Load data is added to the load sensing queue in the order of timestamps. The number of loads in the current load sensing queue is determined. If the number of loads is greater than the number of loads that the time window can accommodate, the load data with the longest queuing time is removed from the sensing queue.

[0012] According to a preferred embodiment of the present invention, the write load intensity calculation method includes:

[0013]

[0014] Where H write Indicates write load strength, w k f represents the weighting coefficient for the k-th load, and n represents the total number of loads within the time window; k This indicates the type of the k-th load, where f is the corresponding value when the k-th load is a write load. k The value is 1, otherwise f k The value is 0; where Furthermore, the weighting coefficient w k The configuration method includes: the load data is divided into different groups according to the percentage of the total load, starting from the nearest current time, and the load data of each group is assigned a weighting coefficient value from large to small according to the time series from the nearest to the farthest group.

[0015] According to a preferred embodiment of the present invention, after obtaining the load data and calculating the write load intensity, the maximum inertia layer is further calculated. The method for calculating the maximum inertia layer is as follows:

[0016]

[0017] Where t represents the maximum inert layer, H is the calculated write load intensity, L is the maximum layer height of the LSM-Tree key-value index hierarchical structure, and T is the fan-out factor (size ratio).

[0018] According to a preferred embodiment of the present invention, the method of adjusting the storage index balancing strategy and tiering strategy to adapt to the write load intensity using the maximum inertia layer specifically includes:

[0019] The maximum lazy layer t is defined as the layer number of the lowest layer using the layering strategy; for all layers above the maximum lazy layer, i.e., layers with layer numbers less than t, the maximum number of sorted segments allowed is T, i.e., the layering strategy is adopted; while for all layers below the maximum lazy layer, i.e., layers with layer numbers greater than t, the maximum number of sorted segments allowed is 1, i.e., the balanced strategy is adopted.

[0020] When t=0, it is a global leveling strategy;

[0021] When t = L, it is a global tiering strategy;

[0022] By adjusting the maximum lazy layer, the single-point query performance and write performance of key-value indexes will vary between the global leveling strategy and the global tiering strategy, thereby improving the single-point query performance or write performance of the index.

[0023] According to a preferred embodiment of the present invention, the configuration of multiple Bloom filter units segmented to generate a Bloom filter unit group specifically involves dividing a single Bloom filter of length n into k Bloom filters of length n. The filter is configured, and a set of corresponding hash functions are allocated to the split Bloom filters according to the optimal hash function allocation method. The split Bloom filters constitute a Bloom filter unit group. The Bloom filter unit group includes a main Bloom filter unit and multiple auxiliary Bloom filter units, and the main Bloom filter unit and the auxiliary Bloom filter units are configured with corresponding per-key lengths.

[0024] According to a preferred embodiment of the present invention, a method for quantitatively calculating the number of active auxiliary units in a Bloom filter unit group defines the number of active auxiliary Bloom filter units as a bit length exponent, and the method for calculating the bit length exponent includes:

[0025]

[0026] Where p is the bit length exponent to be calculated, H is the write load intensity, and U is the number of auxiliary Bloom filter units. The bit length exponent value calculated based on the write load intensity corresponds to the number of auxiliary Bloom filter units that are enabled. The bit length exponent is a rounded value, and the rounded value corresponds to the number of auxiliary Bloom filter units that are enabled.

[0027] According to a preferred embodiment of the present invention, the incremental adjustment of the index state is performed only during querying and writing, and only one layer is adjusted at a time, hence it is an incremental adjustment; if there are multiple layers that need to be adjusted, only the lowest layer is adjusted; the specific adjustment method for each layer is as follows: after calculating the maximum lazy layer, the actual state and expected strategy of each layer are determined according to the current maximum lazy layer, the expected strategy includes a leveling strategy and a tiering strategy, and the actual state includes a leveling state and a tiering state; during querying and writing, if the predicted strategy and the actual state of the current layer are consistent, the adjustment is not performed; if the actual state of the current layer is a leveling state, and the expected strategy of the current layer is a tiering strategy, the current layer is merged during merging and appended to the next layer; if the actual state of the current layer is a tiering state, and the expected state of the corresponding current layer is a leveling strategy, only the current layer is merged; during the query process, all SST files involving the query interval of all layers are scanned, and when there are multiple SST files with overlapping intervals, an intra-layer merge operation is performed.

[0028] According to a preferred embodiment of the present invention, the dynamic adjustment of the number of Bloom filter group units loaded into the index memory, thereby dynamically adjusting the total index memory space occupied by the Bloom filter group and achieving dynamic optimization of memory efficiency, specifically involves: enabling or disabling auxiliary Bloom filter units through memory allocation; generating a full-length Bloom filter group on disk; determining the number of enabled auxiliary filters in memory according to the bit length exponent calculation method; and during the key-value query and write phases, determining the number of enabled auxiliary filters, loading a specified number of auxiliary filters into memory, and simultaneously reducing or increasing the proportion of Bloom filter memory allocated in the total index memory according to a corresponding ratio.

[0029] Another objective of this invention is to provide an adaptive tuning system for LSM-Tree key-value indexes, wherein the system executes the adaptive tuning method for LSM-Tree key-value indexes.

[0030] Another objective of this invention is to provide a computer-readable storage medium storing a computer program that is executed by a processor to implement the adaptive tuning method for the LSM-Tree key-value index.

[0031] Compared with the prior art, the present invention has at least the following beneficial effects:

[0032] The method and system described in this invention incorporate a load-aware module within the index. This load-aware module dynamically acquires load data from the input side and dynamically adjusts the index structure based on the load data, thereby achieving dynamic single-point query and write index resource optimization allocation.

[0033] After obtaining the load information, the method and system of this invention calculate the maximum lazy layer based on the load information. By adjusting the maximum lazy layer, the index can achieve data balancing and data tiering status adjustments across different data layers. Furthermore, based on the actual load information, query-to-write ratio, or write load intensity value, the data balancing and data tiering status are adjusted to match the query-to-write ratio, thus improving overall data throughput performance. The maximum lazy layer defined in this invention is the layer number of the lowest layer implementing the tiering strategy, i.e., the layer number of the last tiering strategy. Different layer leveling and tiering strategies are configured based on the maximum lazy layer and the write load intensity in the received load data, thereby improving the adaptability of storage and read index resources. Based on the aforementioned maximum lazy layer, the Bloom filter unit, which takes the input load intensity and sharding as input, dynamically executes the aforementioned different layer leveling and tiering strategies, allowing the LSM-Tree key-value index to dynamically adjust its global leveling and tiering strategies according to the input load intensity.

[0034] The method and system described in this invention utilize a segmented Bloom filter. By constructing a bit length index containing multiple different Bloom filter units, the bit length index reflects the number of Bloom filter units that are enabled after the segmentation. The bit length index is dynamically configured according to the different write negative intensities at the input end so that the enabled Bloom filter units are adapted to the write load intensity at the input end, thereby improving the adaptability of reading storage resources and writing resources.

[0035] When the method and system of the present invention obtain the corresponding request data using the load sensing module, the load sensing module is used to obtain the timestamp of the request. Based on the timestamp and the current time, it is determined whether the current request has expired, and expired requests are removed, thereby ensuring the real-time performance of the queue in the load sensing module. Attached Figure Description

[0036] Figure 1 The diagram shown is a flowchart of an adaptive tuning method for LSM-Tree key-value indexes according to the present invention.

[0037] Figure 2 The diagram shown is a schematic representation of the overall structure of the LSM-Tree in this invention.

[0038] Figure 3 The diagram shown illustrates the working principle of a flow-based load sensing module.

[0039] Figure 4The diagram shows the working principles of two merging strategies (leveling and tiering).

[0040] Figure 5 This is a schematic diagram of inter-layer merging.

[0041] Figure 6 This shows a schematic diagram of intra-layer merging.

[0042] Figure 7 The diagram shown is a schematic of a Bloom filter unit group implemented in slices.

[0043] Figure 8 This diagram illustrates the relationships when embedded in the storage index.

[0044] Figure 9 The diagram shows the throughput before and after optimization under high read load.

[0045] Figure 10 The diagram shows the throughput before and after optimization under high write load. Detailed Implementation

[0046] The following description is intended to disclose the present invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art. The basic principles of the invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the invention.

[0047] It is understood that the term "a" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple, and the term "a" should not be understood as a limitation on the number.

[0048] Please refer to Figures 1-10 This invention discloses an adaptive tuning method and system for LSM-Tree key-value indexes, wherein the method mainly includes the following steps:

[0049] Configure a load awareness module at the index data receiving end, obtain load data through the load awareness module, and calculate the write load intensity in the load data;

[0050] The maximum lazy layer is calculated and updated based on the write load intensity. Based on the calculated write load intensity, the expected merging strategy of each key-value index layer adapted to the write load intensity is adjusted using the maximum lazy layer.

[0051] Configure multiple Bloom filter units for fragmentation, generate Bloom filter unit groups, calculate the bit length index of the Bloom filter unit groups for fragmentation, and control the number of Bloom filter units enabled based on the bit length index of the Bloom filter unit groups and the adaptation relationship of the calculated write load.

[0052] Based on the expected merging strategy adapted to the write load intensity, the index status is gradually adjusted during index entry queries and writes to achieve dynamic optimization of the index status.

[0053] Based on the calculated number of Bloom filter units enabled, the number of Bloom filter units loaded into the index memory is dynamically adjusted during index entry querying and writing, thereby dynamically adjusting the total index memory space occupied by the Bloom filter group and achieving dynamic optimization of memory efficiency.

[0054] Specifically, in order to better illustrate the technical effects of the present invention, the present invention is further explained as follows to explain the technical effects and logic of the present invention:

[0055] like Figure 2 As shown, an LSM-Tree key-value index is an ordered index data structure, typically composed of multiple ordered sub-components distributed across memory and disk, arranged hierarchically from top to bottom. Generally, LSM-Tree-based index implementations store data in memory and external storage. The memory portion serves as the write and query cache for the entire index, located at the top layer of the hierarchical structure. The memory portion is usually presented as an ordered in-memory data structure, such as a red-black tree or skip list. The external storage portion is presented as multiple ordered files (usually called SST files) maintained by size, arranged hierarchically according to their update time. The memory portion is generally called the memory table layer, and the other external storage layers are named L0, L1, L2, and so on, from top to bottom, up to Ln. Each layer, including the memory table layer, has a size limit; when the limit is reached, a merging with the next layer is triggered. Therefore, LSM-Trees exhibit the characteristic that the data volume is smaller and newer at higher levels, while the data volume is larger and older at lower levels.

[0056] When writing key-value pairs, LSM-Tree typically inserts the corresponding key-value pairs directly into the memory table and immediately returns a successful write. The process of key-value pairs moving from the memory table to disk and then to the lower-level SST (Site-Site Table) is completed in subsequent periodic merge processes, which is the meaning of LSM-Tree's non-in-place update feature. To ensure secure data persistence, LSM-Tree-based key-value indexes incorporate a Write-Ahead Log (WAL) process into the write operation, appending the specific content of the write operation to the recovery log on disk before writing it into the memory table.

[0057] like Figure 3 The diagram illustrates the principle of the load-aware module. The index achieves load awareness by statistically analyzing the proportion of single-point query and write requests within a time window. First, in the dynamically tuned key-value index, a stream processing module (i.e., the load-aware queue) runs synchronously with the main index. All query and write requests received by the key-value index are sent to this stream processing module during execution. Within the stream processing module, a queue of a pre-defined size is maintained. All received requests are stored in this queue according to their logical processing order. When a new request enters, the oldest request at the end of the queue is evicted, ensuring that the requests within the logical time window maintained by the stream processing module reflect the index load trend over a recent period. Since the load within the key-value index consists only of write load and single-point query load, this paper uses write load intensity as an indicator to represent the load status within the index. When write load intensity is low, the load within the index is considered to be biased towards single-point query load; when write load intensity is high, the load within the index is considered to be biased towards write load.

[0058] This article uses a weighted average method to estimate the write load intensity within the current index:

[0059]

[0060] Among them, H write To write load strength, w k The weights corresponding to the load intensity k are given, where n is the total number of loads in the sensing queue within the time window, and f is the weight. k This represents the type value corresponding to the load strength, where w1+w2+…+w n =1. For each request, if it is a write request, then f=1; otherwise, f=0. The method of writing load intensity shows that requests closer to the current time have a greater impact on load estimation, while past requests have a smaller impact on current load estimation. Therefore, this invention assigns values ​​to the above weights, selects the load data closest to the current time point, groups them according to the total number of load data within the time window, and assigns values ​​to the load data in each group from the most recent to the furthest group according to the time series, from largest to smallest. For example, in this invention, for the above weight w... k The first 10% of load data groups closest to the current time point are assigned a weight sum of 1 / 2, while the second 10% of load data groups are assigned a weight sum of 1 / 4. This process is repeated for each load data group until the sum of the weights of all load data groups equals 1. The specific steps for calculating the load intensity include:

[0061] S01, Before the indexing process begins, a specific size is set as the logical time window size for the sensing queue;

[0062] S02, when the index receives any load, add the load to the head of the queue in parallel and execute the load;

[0063] S03, If the number of loads stored in the queue exceeds the window size, discard the old loads at the tail of the queue;

[0064] S04 calculates the current write load intensity in real time using a weighted summation and averaging method;

[0065] S05, Update the write load intensity for the next lifetime stored in the index;

[0066] S06. If the load timestamp stored at the tail of the queue exceeds a certain range of the current time, then all timed-out loads at the tail of the queue will be eliminated to avoid outdated data affecting the current load estimate.

[0067] To avoid the influence of cold data on the estimation results, requests that might affect the estimation results also need to be discarded. Since storage indexes may experience downtime and restarts, or because the storage index itself is secondary storage or cold storage indexes, the last requests in the streaming window may be operations from hours or even days ago. Statistics on such requests are undoubtedly meaningless for current load awareness. Therefore, each request is added to the awareness queue along with its current timestamp, `currentTime`. Simultaneously, the key-value index maintains a global variable, `lifetime`, as the maximum allowed lifespan for each request. This time is 6 hours by default, but can be set. During load estimation, the index checks the expiration time of each request (`currentTime + lifetime`) and compares it with the current time. If the expiration time of a request is less than the current time, the index considers the request expired, and expired requests are not included in the statistics.

[0068] The impact of merging strategies on index performance:

[0069] There are two common merging strategies in LSM-Trees: leveling and tiering. For example... Figure 4As shown, the core difference between these two strategies lies in the upper limit of the number of sorted runs allowed per level. In the most classic implementation, one sorted run often corresponds to one SST file. However, to reduce the impact of write amplification, current mainstream implementations often generate multiple SST files for each sorted run. While the data within each SST file is ordered, the order between SST files at the same level is not necessarily true. A sorted run is an abstract concept composed of a set of SST files with non-overlapping ranges. It can be seen that the number of sorted runs at a certain level reflects the order relationship between the SST files at that level. The Leveling strategy merges all SST files at the current level with all SST files at the next level that involve this range, resulting in only one sorted run per level after merging. The Tiering strategy, on the other hand, does not involve data at the next level during merging; it only merges all SST files at the current level and then directly inserts them into the next level. Therefore, it can be summarized that the core difference between the Leveling and Tiering strategies lies in their requirements regarding the upper limit of sorted runs at each level. The leveling strategy requires that each level can have only one sorted segment, while the tiering strategy requires that each level can have at most T sorted segments.

[0070] In this invention, the concept of a maximum lazy layer is defined to describe the currently used merging strategy model. The maximum lazy layer t is defined as the layer number of the lowest layer using the tiering strategy. For all layers less than t (i.e., layers above), the maximum allowed number of sorted segments is T, i.e., the tiering strategy is adopted; while for all layers greater than t (i.e., layers below), the maximum allowed number of sorted segments is 1, i.e., the leveling strategy is adopted. Therefore, the following relationship exists:

[0071] When t=0, it is a global leveling strategy;

[0072] When t = L, it is a global tiering strategy;

[0073] By adjusting the maximum lazy layer, the single-point query performance and write performance of key-value indexes will vary between the global leveling strategy and the global tiering strategy, thereby improving the single-point query performance or write performance of the index.

[0074] Merging strategy is dynamically adjusted:

[0075] By adjusting the index merging strategy, the ratio of index query performance to write performance can be matched to the current write load intensity of the index. The following calculation method is used to calculate the maximum lazy layer t of the key-value index at runtime, where H is the estimated write load intensity, L is the maximum layer height, and T is the fan-out coefficient:

[0076]

[0077] Since the aforementioned time window is dynamic, this invention requires dynamic adjustment of parameters according to the above formula:

[0078] S201. Calculate the global maximum inertia level t based on the write load intensity;

[0079] S202, Update the value of the global maximum lazy layer;

[0080] S203, calculate the global Bloom filter bit length exponent p based on the write load intensity;

[0081] S204: Calculate memory table allocation based on the new bit length exponent p, and expand or shrink the memory table accordingly.

[0082] S205, Update the global Bloom filter bit length index p1.

[0083] The parameter update process described above (S201-S205) illustrates the update process of the required maximum lazy layer and bit length exponent p parameter, realizing the adaptive optimal allocation of storage index resources for data query and write loads by adjusting memory table allocation according to load intensity.

[0084] When a merge is triggered, the index aggregates all SST files in the current layer to generate an iterator. Then, based on the read maximum lazy layer and the layer number of the next layer, it determines the merge strategy to be used when merging to the next layer. If the actual state of the next layer is consistent with the merge strategy set for that layer, then the merge can be performed directly according to the set merge strategy. If they are inconsistent, there are two cases:

[0085] 1. The next level currently has only one sorted segment (leveling state), but the merging strategy used is tiering. In this case, the data in the current level can be directly merged and then inserted into the next level. This is because the tiering state has more lenient write requirements, so no additional processing is needed.

[0086] 2. The next level currently has multiple sorted segments (tiering state), but the merging strategy used is leveling. In this case, the index will attempt to reduce the number of sorted segments. First, the index determines the key range of the current level, then searches for all SST files in the next level that overlap with the key range of the current level, and includes them in the iterator used for this merge. In this way, if a level changes from a tiering strategy to a leveling strategy, after multiple such merges, the number of sorted segments will gradually decrease until it becomes 1. At this point, the level has reached the expected leveling state. However, if the next level has no or only one SST file that overlaps with the key range of the current level, then this merge process will neither increase nor decrease the number of sorted segments in the next level. This process is called inter-level merging. Figure 5 As shown. The specific merging steps are as follows:

[0087] S301, during the execution lifecycle, calculate the maximum lazy layer based on the current write load intensity;

[0088] S302, when a merge is triggered, the merge is performed according to the merge strategy of the next level and the actual situation;

[0089] S303: If the expected strategy and the actual strategy match, the merger shall be implemented directly according to the actual strategy; if they do not match, adjustments shall be made as appropriate.

[0090] S3031, if the next layer is in a leveling strategy and the expected strategy is a tiering strategy;

[0091] S3032 directly merges the current layer and then appends it to the next layer;

[0092] S3033, if the next layer is in a tiering strategy, the expected state is a leveling strategy;

[0093] S3034, Scan additional sorted segments and merge any sorted segments that overlap with the current interval;

[0094] S304, When making a query request, check the expected strategy and actual state of the current layer and add additional merge operations;

[0095] S3041, During the query, record all sorted segments that involve the current query value;

[0096] S3042, If the number is greater than 1, then an intra-layer merge needs to be performed on these sorted segments;

[0097] S305, complete the current merge or query operation.

[0098] The above S301 to S305 operation procedures illustrate the merging process when the actual state and expected state of the next layer below the maximum lazy layer are the same or different. This can enable the entire key-value index to be adapted to the leveling and tiering strategies of different layers based on the query-write ratio of the load data.

[0099] To avoid situations where only query queries exist and merging is never triggered, the index will attempt additional merging operations during the query process. During a single-point query, the index determines whether to query the current file based on the data range in the SST file metadata. In the leveling state, since the ranges of SST files do not overlap, at most one SST file in a sort segment needs to be accessed; however, in the tiering state, the sort segments may overlap, and in the worst case, all SST files in all sort segments need to be accessed. If the current layer's state needs to be adjusted from tiering to leveling, it indicates that the index's load preference is shifting towards query load. In this case, there will be more query operations, so distributing the architecture across queries is reasonable. First, the index still needs to compare the current layer number with the value of the maximum lazy layer to determine the strategy adopted by the current layer. After this:

[0100] 1. If the merging strategy of the current layer is the tiering strategy, then during the query, all SST files that may contain the value to be queried will be queried normally, without any additional processing;

[0101] 2. If the current layer's merge strategy is leveling, then while querying all SST files that might contain the query value, the number of SST files whose intervals involve the query value is recorded. After the query for the current layer is completed, if the number of SST files with overlapping intervals is greater than 1, it indicates that there is interval overlap in this layer, and the current layer is marked as potentially requiring intra-layer merging. After a query is completed, the index will trigger an intra-layer merge with the current key as the range for the last layer marked as requiring intra-layer merging. If the current layer has been adjusted to the leveling strategy, then at most one SST file's interval will contain the query value, and a merge will not be triggered; otherwise, if there are multiple SST files whose intervals contain the query key, they will be merged to reduce the number of sorted segments. The main reason for triggering a merge only for the last layer that needs merging is that triggering too many merges at once may affect the performance of subsequent operations. When only one layer is selected to trigger a merge, triggering intra-layer merging for lower layers provides a higher overall performance benefit to the index. Intra-layer merging is the process of merging only SST files with overlapping intervals in the current layer, such as... Figure 6 As shown.

[0102] The impact of Bloom filters on index performance:

[0103] Bloom filters are part of the SST file metadata and share memory with in-memory tables. When the available memory for the key-value index is fixed, the ratio of memory allocated to in-memory tables to that allocated to Bloom filters impacts system read and write performance. Allocating more memory to in-memory tables reduces write latency, improving write performance; conversely, increasing the bit length of the in-memory Bloom filter reduces false positives, improving single-point query performance. If the total number of false positives for single-point queries at any given time is expected to be similar, then the number of false positives should be lower when there are many query requests, while a higher number of false positives can be tolerated when there are fewer query requests. Since the number of false positives is directly related to the bit length of the Bloom filter, if the key-value index can predict the proportion of query and write requests over a future period, the bit length of the Bloom filter can be adjusted to the desired length based on this proportion, allowing limited memory to better serve the overall system throughput.

[0104] This invention further provides a method for dynamically adjusting Bloom filters, wherein the bit length allocation of the Bloom filter is usually fixed because, in principle, the keys inserted into the Bloom filter are almost impossible to delete. Therefore, to reallocate the bit length of the Bloom filter, it is usually only possible to rebuild the Bloom filter. However, if the filter needs to be rebuilt every time it is adjusted, the overhead is far from acceptable. In this invention, a method for approximately dynamically adjusting the bit length of the Bloom filter can be achieved through a piecewise approach. The piecewise approach specifically includes: splitting the original filter vector of length n into k filter vectors of length n. The filter vector is used to distribute the original set of hash functions evenly across each smaller filter in an optimal hash function allocation manner. Existing research has shown that for a Bloom filter with a bit-per-key length of p, allocating ln2*p hash functions achieves optimal accuracy. The optimal hash function allocation method described in this invention distributes the data evenly in this way. Each small Bloom filter after this method of splitting is called a Bloom filter unit, and a large Bloom filter composed of multiple filter units is called a Bloom filter unit group. Considering that with uniform distribution, the size of a single filter unit in the index may be insufficient to independently perform the most basic filtering function,

[0105] In one preferred embodiment of the invention, the filter group in this index is divided into one main filter unit and four auxiliary filter units. The main filter unit is configured with 10 bits per key, serving as a basic filter unit that is never turned off; the auxiliary filter units are configured with 2 bits per key and are only enabled when the system is heavily loaded with queries. The structure of the sharded Bloom filter unit group is as follows: Figure 7 As shown. The above grouping method is merely an example of the present invention, and the present invention does not limit the grouping method of the main filtering unit and the auxiliary filtering unit.

[0106] To describe the per-key bit length currently allocated by the index for the Bloom filter, this invention uses the concept of a bit length exponent. Its value represents the number of secondary filter units currently enabled by the index. In adjusting the bit length exponent, since a longer Bloom filter is loaded by the index, the false positive rate of a single query is lower. Therefore, when there are fewer query requests, the index can tolerate the additional query overhead caused by a higher false positive rate in exchange for improved index write performance. Specifically, the index will adjust the number of enabled secondary filter units based on the write load intensity. The specific calculation of the bit length exponent p is as follows:

[0107]

[0108] Where p is the bit length exponent, H is the write load intensity, and U is the number of auxiliary Bloom filter units. In this invention, the bit length exponent is rounded down to calculate the total number of auxiliary Bloom filter units that need to be activated. When the number of auxiliary Bloom filter units is 4, the following relationship can be obtained:

[0109] When the write load intensity is less than 0.1 (i.e., almost entirely query load), the calculated bit length exponent is greater than 4. Rounding this up, the bit length exponent p is 4, and all auxiliary filter units need to be enabled. When the write load intensity is between 0.1 and 0.2, the bit length exponent p calculated using the above formula is 3, so one auxiliary unit needs to be disabled. When the write load intensity is between 0.2 and 0.3, the bit length exponent p calculated using the above formula is 2, so two auxiliary units need to be disabled. And so on. When the write load intensity is greater than 0.4, the bit length exponent p calculated using the above formula is 0, so all auxiliary units need to be disabled, and only the main filter unit is used. It is important to emphasize that enabling a filter unit means that this filter unit will be loaded into memory and used to determine the presence of a key. Even if a filter unit is disabled, it still exists in the SST file. This also means that when constructing a new SST file, it is still necessary to generate a complete Bloom filter unit group containing all the keys in this SST file. The process of generating a new SST file and constructing all Bloom filter units is as follows: Figure 8 As shown.

[0110] It should be noted that, regarding the method for adjusting the position length of the Bloom filter, the present invention proposes the following specific steps:

[0111] S401, when a single point query on a key is triggered, a pre-query is performed based on the Bloom filter of the SST in which it is located;

[0112] S402, check in memory whether the corresponding filter has been loaded;

[0113] S403, if the corresponding Bloom filter does not exist, load the corresponding number of Bloom filter units according to the bit length exponent;

[0114] S4031, if the memory space of the Bloom filter is insufficient, the filter that is already in memory will be evicted in the LRU (Least Recently Used Page Replacement) manner;

[0115] S404 If the corresponding filter has been loaded, but the loaded length is less than the bit length exponent requirement, then while it is currently in use, asynchronously update this filter to the corresponding cell number;

[0116] S405: If the corresponding filter has been loaded, but the loading length exceeds the bit length exponent requirement, no processing is performed.

[0117] S406, complete the key check operation.

[0118] The steps S401 to S406 above implement the methods for finding Bloom filters, calculating bit length exponents, and adjusting the function of corresponding Bloom filter units.

[0119] Figure 9 , 10 The graph shows the test results comparing the optimized and unoptimized indexing methods. Higher throughput indicates better indexing performance.

[0120] The processes described above with reference to the flowcharts in the embodiments disclosed in this invention can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication component, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), it performs the functions defined in the methods of this application. It should be noted that the computer-readable medium described above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wire segments, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical fibers, RF, etc., or any suitable combination thereof.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0122] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are merely examples and do not limit the present invention. The purpose of the present invention has been fully and effectively achieved, and the functions and structural principles of the present invention have been shown and explained in the embodiments. Without departing from the stated principles, the implementation of the present invention may have any variations or modifications.

Claims

1. An adaptive tuning method for LSM-Tree key-value indexes, characterized in that, The method includes: Configure a load awareness module at the index data receiving end, obtain load data through the load awareness module, and calculate the write load intensity in the load data; The maximum lazy layer is calculated and updated based on the write load intensity. Based on the calculated write load intensity, the expected merging strategy of each key-value index layer adapted to the write load intensity is adjusted using the maximum lazy layer. The method for calculating the maximum inert layer is as follows: ; Where t represents the maximum inert layer, H is the calculated write load intensity, L is the maximum layer height of the LSM-Tree key-value index hierarchical structure, and T is the fan-out coefficient; The storage index balancing strategy and tiering strategy that utilizes the maximum lazy layer to adapt to the write load intensity are specifically as follows: The maximum lazy layer t is defined as the layer number of the lowest layer using the layering strategy; for all layers above the maximum lazy layer, i.e., layers with layer numbers less than t, the maximum number of sorted segments allowed is T, i.e., the layering strategy is adopted; while for all layers below the maximum lazy layer, i.e., layers with layer numbers greater than t, the maximum number of sorted segments allowed is 1, i.e., the balanced strategy is adopted. When t=0, it is a global leveling strategy; When t=L, it is a global tiering strategy; By adjusting the maximum lazy layer, the single-point query performance and write performance of key-value indexes will vary between the global leveling strategy and the global tiering strategy, thereby improving the single-point query performance or write performance of the index. Configure multiple Bloom filter units for fragmentation, generate Bloom filter unit groups, calculate the bit length index of the Bloom filter unit groups for fragmentation, and control the number of Bloom filter units enabled based on the bit length index of the Bloom filter unit groups and the adaptation relationship of the calculated write load. The configuration of multiple Bloom filter units into a Bloom filter unit group specifically involves dividing a single Bloom filter of length n into k Bloom filter units of length n. The filter is configured, and a set of corresponding hash functions are allocated to the split Bloom filters according to the optimal hash function allocation method. The split Bloom filters constitute a Bloom filter unit group. The Bloom filter unit group includes a main Bloom filter unit and multiple auxiliary Bloom filter units, and the main Bloom filter unit and the auxiliary Bloom filter units are configured with corresponding per-key lengths. Based on the expected merging strategy adapted to the write load intensity, the index status is gradually adjusted during index entry queries and writes to achieve dynamic optimization of the index status. Based on the calculated number of Bloom filter units enabled, the number of Bloom filter units loaded into the index memory is dynamically adjusted during index entry querying and writing, thereby dynamically adjusting the total index memory space occupied by the Bloom filter group and achieving dynamic optimization of memory efficiency.

2. The adaptive tuning method for LSM-Tree key-value indexes according to claim 1, characterized in that, The load awareness module is configured at the data receiving end of the LSM-Tree key-value index. The load awareness module includes a load awareness queue, which is used to acquire load data from the input side of the LSM-Tree key-value index. The load data includes write load data and read load data. Specifically, a time window of a preset size is configured for the load sensing module, and a load sensing queue is constructed. Load data is added to the load sensing queue in the order of timestamps. The number of loads in the current load sensing queue is determined. If the number of loads is greater than the number of loads that the time window can accommodate, the load data with the longest queuing time is removed from the sensing queue.

3. The adaptive tuning method for LSM-Tree key-value indexes according to claim 2, characterized in that, The method for calculating write load intensity includes: ； in Indicates write load strength. This represents the weighting coefficient for the k-th load, and n represents the total number of loads within the time window; This indicates the type of the k-th load, where when the k-th load is a write load, the corresponding... The value is 1, otherwise The value is 0; where =1.

4. The adaptive tuning method for LSM-Tree key-value indexes according to claim 3, characterized in that, The weighting coefficients The configuration method includes: the load data is divided into different groups according to the percentage of the total load, starting from the nearest current time, and the load data of each group is assigned a weighting coefficient value from large to small according to the time series from the nearest to the farthest group.

5. The adaptive tuning method for LSM-Tree key-value indexes according to claim 1, characterized in that, A quantitative calculation method for the number of active auxiliary units in a Bloom filter unit group is provided, defining the number of active auxiliary Bloom filter units as the bit length exponent. The calculation method for the bit length exponent includes: ; Where p is the bit length exponent to be calculated, H is the write load intensity, and U is the number of auxiliary Bloom filter units. The bit length exponent value calculated based on the write load intensity corresponds to the number of auxiliary Bloom filter units that are enabled. The bit length exponent is a rounded value, and the rounded value corresponds to the number of auxiliary Bloom filter units that are enabled.

6. The adaptive tuning method for LSM-Tree key-value indexes according to claim 1, characterized in that, The incremental adjustment of the index state is performed only during queries and writes, adjusting only one layer at a time, hence the term incremental adjustment. If multiple layers need adjustment, only the lowest layer is adjusted. The specific adjustment method for each layer is as follows: After calculating the maximum lazy layer, the actual state and expected strategy of each layer are determined based on the current maximum lazy layer. The expected strategy includes a leveling strategy and a tiering strategy, and the actual state includes a leveling state and a tiering state. During queries and writes, if the predicted strategy and actual state of the current layer are consistent, the adjustment is not performed. If the actual state of the current layer is a leveling state, and the expected strategy of the current layer is a tiering strategy, the current layer is merged during merging and appended to the next layer. If the actual state of the current layer is a tiering state, and the expected state of the corresponding current layer is a leveling strategy, only intra-layer merging is performed on the current layer. During the query process, all SST files involving the query interval are scanned. When multiple SST files with overlapping intervals exist, an intra-layer merging operation is performed.

7. The adaptive tuning method for LSM-Tree key-value indexes according to claim 1, characterized in that, The dynamic adjustment of the number of Bloom filter group units loaded into the index memory, thereby dynamically adjusting the total index memory space occupied by the Bloom filter group and achieving dynamic optimization of memory efficiency, specifically involves: enabling or disabling auxiliary Bloom filter units through memory allocation; generating a full-length Bloom filter group on disk; determining the number of enabled auxiliary filters in memory according to the bit length exponent calculation method; and during the key-value query and write phases, loading a specified number of auxiliary filters into memory based on the determined number of enabled auxiliary filters, while simultaneously reducing or increasing the proportion of Bloom filter memory allocated in the total index memory according to the corresponding ratio.

8. An adaptive tuning system based on LSM-Tree key-value index, characterized in that, The system executes the adaptive tuning method of the LSM-Tree key-value index as described in any one of claims 1-7.

9. A computer-readable storage medium storing a computer program that is executed by a processor to implement the adaptive tuning method for an LSM-Tree key-value index as described in any one of claims 1-7.

Images