A system that uses learned indexes to accelerate read operations in a key-value storage system

Abstract

This invention discloses a system for accelerating read operations in a key-value storage system using a learned index, comprising a key space partitioning module, a learned index module, a data writing module, a data organization module, an index table update module, an index training module, a garbage collection module, a data retrieval module, a data reading module, and a dynamic index table module; the invention aims to improve the read performance of the system on high-speed storage devices.

Description

Technical Field

[0001] This invention relates to the field of data reading, and in particular to a system that uses a learned index to accelerate read operations in a key-value storage system. Background Technology

[0002] In data-intensive applications, efficient data access is crucial. Traditional solutions often rely on classic index structures, such as skip lists and B+ trees, which can achieve data retrieval in logarithmic time complexity O(log n). While these general-purpose data structures perform well in many situations, they are not optimized for specific data distributions. The limitation of traditional indexes lies in their inherent design, which fails to fully utilize the potential patterns in data distribution. However, data often follows certain distribution patterns, and if these patterns are effectively identified and utilized, the efficiency of indexing can be greatly improved. With the powerful capabilities of machine learning, we can gain a deep understanding of the distribution characteristics of data and build predictive models accordingly. Learning indexes are an application of this concept; they analyze data distribution and use machine learning algorithms to predict the location of data indexes. Compared to traditional indexes, learning indexes can not only describe the distribution of data but also predict the location of data with specific distribution characteristics, thereby achieving data access in constant time complexity (O(1)).

[0003] In modern key-value stores, LSM-trees (Log Structure Merge Trees) have become a core component of the infrastructure due to their excellent write performance and efficient space utilization. However, the efficiency of read operations is equally crucial, as it directly impacts the database system's response time and user experience. An efficient read process ensures that data can be quickly retrieved and made available to applications, which is especially important for analytical tasks and interactive operations requiring large amounts of data reading. The read operation flow of an existing LSM-tree key-value store is as follows:

[0004] 1. Find files: This step involves identifying the potential set of files containing the target key. Based on the range of keys, the system traverses the LSM-tree hierarchy to identify data files that may contain key-value pairs.

[0005] 2. Loading Index Blocks and Filter Blocks (loadIB+FB): Once the relevant files are identified, the system loads the index blocks (IB) and filter blocks (FB) for these files. The index blocks contain metadata indicating the location of the key in the data file, while the filter blocks are used to quickly determine whether a key exists in the file.

[0006] 3. Search the index block (searchIB): Next, a search is performed within the index block to locate the exact position of the data. This typically involves performing a binary search or other efficient search algorithm on the index block.

[0007] 4. Search Filter Block (searchFB): At the same time, the filter block is queried to determine whether the key-value pair may not exist in the data file, thereby achieving fast negative queries and avoiding unnecessary data block loading.

[0008] 5. Loading Data Blocks (loadDB): Once the index block indicates the location of the data, the system will load the data block (DB) containing the actual key-value pairs.

[0009] 6. Search Data Block (searchDB): Next, a search operation is performed within the data block to retrieve specific key-value pairs. Since data blocks are typically small and may be stored on fast storage media, this step can be completed quickly.

[0010] 7. Read Value: Finally, once the key-value pair is found, the system reads the value and returns it to the requester. This step completes the key-to-value retrieval process and is the final step in the entire read operation.

[0011] However, with advancements in storage technology, especially the adoption of high-speed storage devices such as solid-state drives (SSDs) and non-volatile memory (NVMe), a new bottleneck has emerged: the proportion of index lookup time in the overall read operation time has been gradually increasing. This is because while these high-speed storage devices significantly reduce the time spent loading data from disk, the time spent on index lookup—especially in large-scale and complex LSM-tree structures—has not decreased significantly. Since LSM-trees use index blocks as the index granularity, each read operation requires multiple steps for index lookup, including finding files, loading index blocks and filtering blocks (loadIB+FB), searching index blocks (searchIB), and searching filtering blocks (searchFB). Furthermore, the time cost of most of these steps is not reduced by the increased read / write speeds of storage devices. Therefore, index lookup becomes a key factor limiting the overall read operation performance. Summary of the Invention

[0012] To address the existing problems, this invention provides a system that uses a learned index to accelerate read operations in a key-value storage system. The specific solution is as follows:

[0013] A system that uses a learned index to accelerate read operations in a key-value storage system includes a key space partitioning module, a learned index module, a data writing module, a data organization module, an index table update module, an index training module, a garbage collection module, a data retrieval module, a data reading module, and a dynamic index table module.

[0014] The key space partitioning module divides the entire key space into multiple consecutive intervals—buckets—each responsible for storing key-value pairs within a certain range. The learning index module locates buckets and learns and adjusts the position information of key-value pairs. The data writing module writes data into the hash buffer of the target bucket determined by the learning index. The data organization module sorts the data in the hash buffer after the data reaches a certain amount and writes it to disk in the form of an ordered string table. The index table update module dynamically updates the learning index. The index training module retrains the learning index within the bucket after the index table is updated to ensure it accurately reflects the latest distribution of key-value pairs. The garbage collection module performs intelligent garbage collection to maintain storage efficiency and system performance. The data retrieval module retrieves data from the learning index in the target bucket. The data reading module reads the required data from the disk based on the position information provided by the index after retrieving the data. The index table is stored on the disk, and the dynamic index table module dynamically loads active index content into memory through a global translation dictionary. This method provides performance comparable to a fully in-memory index table while maintaining low memory usage.

[0015] Preferably, the learning index module includes a positioning module and an adjustment module; the positioning module is a learning index module based on model insertion, and the adjustment module is a learning index module based on cache insertion.

[0016] Preferably, in the positioning module, the model insertion operation is based on the current distribution pattern of the bucket; specifically, node i in the model represents the node at position i, which is responsible for indexing all key values ​​whose model calculation results fall within the interval [i, i+1); in the model, nodes are divided into two states: unused nodes and nodes in use; unused nodes point to the previous node in use, while nodes in use indicate the range of all key-value pairs from the current node to the next node in use;

[0017] When the bucket undergoes structural changes, such as splitting or merging, the model of the present invention performs insertion or deletion operations accordingly; these operations require special care to ensure that all unused nodes between the inserted or deleted node and the next node in use correctly point to the new node in use.

[0018] Preferably, in order to resolve potential conflicts when inserting nodes in the model, a local amplification mechanism is adopted. Under this mechanism, any conflicting node will be expanded into multiple nodes to maintain the accuracy and efficiency of the model.

[0019] Preferably, the adjustment module is used to obtain the position information of key-value pairs (kv-pairs); in order to process newly inserted data, a hash table is used for temporary buffering, and once the amount of data in the hash table exceeds the set threshold, the system will automatically trigger the retraining of the model.

[0020] Preferably, the waste recycling module implements two waste recycling mechanisms: a combined recycling mechanism and a bin splitting recycling mechanism.

[0021] Preferably, the merging and recycling mechanism begins to function after the index information is updated in the index table; the system first recalculates the proportion of invalid data in the bucket, and when this proportion exceeds a preset threshold, the garbage collection process is triggered; the specific operations include:

[0022] ST1, Data Sorting: All ordered strings in the buckets will be loaded into memory and re-sorted;

[0023] ST2, Data Cleaning: During the sorting process, invalid and garbage data will be marked and removed;

[0024] ST3, Data Rewrite: The cleaned and valid data is rewritten back to the disk;

[0025] ST4, Index Update: Accordingly, the new position information of the key-value pairs in the bucket needs to be updated in the index table.

[0026] Preferably, the bucket splitting operation is performed after the ordered string table has been written to disk; the system will evaluate whether the ratio of the amount of data in the bucket to the average level exceeds a certain threshold. If it does, the splitting process is initiated as follows:

[0027] ST1', Data Sorting and Splitting: Similarly, all ordered string tables in the bucket are first loaded into memory and sorted, and then the data is equally split into two parts;

[0028] ST2', Data Allocation: The first part of the data will be put back into the original bucket, while the second part will be allocated to a newly created bucket;

[0029] ST3', Index Inheritance: The new bucket uses the corresponding part of the index information in the original bucket;

[0030] ST4', Model Adjustment: The two-bucket learning index model does not need to be completely retrained. Since the relative positions of key-value pairs in the index table have not changed, only a few parameters in the model need to be adjusted.

[0031] ST5', Index Insertion: The newly generated bucket will be inserted as a node into the model-based insertion learning index used to locate the bucket.

[0032] Preferably, the dynamic indexing module is designed to create a dynamic index table that achieves low memory consumption while maintaining high performance close to that of a full-memory index table. The system's index table is primarily stored on disk to minimize memory usage. In memory, a cached index table and a global translation dictionary are maintained. When the learning index model initiates a query request, the system first checks if the cached index table contains the required mapping information. If the relevant information is available in the cache, the system will directly provide the required data from the cache, thereby accelerating the reading process. If the requested mapping information is not found in the cached index table, it needs to be retrieved from the index table on disk and loaded into the cache.

[0033] Preferably, considering the limited size of the cache index table, a segmented Least Recently Used (LRU) replacement algorithm is used to manage cached content and optimize the utilization of the cache index table. When the cache is full, the algorithm determines which entries should be evicted to make room for new mapping information. Since the index table and global translation dictionary are processed first in each write operation, the mapping information in the cache can be easily erased without complex additional operations.

[0034] The beneficial effects of this invention are as follows:

[0035] The present invention aims to improve the read performance of the system on high-speed storage devices.

[0036] 1. Innovative Indexing Mechanism: This invention transforms the system's indexing granularity from index blocks to index key-value pairs and creates an ordered index table, laying the foundation for introducing a learning indexing model. Utilizing this ordered index table, learning indexing technology is implemented, significantly improving the efficiency of index lookup during data retrieval by accurately predicting the position of the target key in the index table.

[0037] 2. Reduced compaction requirements. By using indexed key-value pairs, this invention significantly reduces the frequent data compaction operations performed to maintain data order, thereby improving system efficiency and reducing maintenance costs.

[0038] 3. Intelligent Waste Recycling Mechanism: This invention introduces a highly efficient intelligent waste recycling mechanism that combines merging and bin splitting strategies, effectively preventing the unnecessary expansion of storage space. This mechanism ensures optimized utilization of storage space, prevents unnecessary increases in space costs, and thus improves the efficiency and economy of the entire system in storage management.

[0039] 4. Dynamic Index Table: To reduce the memory footprint of the index table, this invention employs the concept of a dynamic index table. The index table is stored on disk, and active index content is dynamically loaded into memory through a global translation dictionary. This method provides performance comparable to a fully in-memory index table while maintaining low memory usage. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a schematic diagram of a method based on the system of the present invention;

[0042] Figure 2 This is a schematic diagram of a learning index based on model insertion.

[0043] Figure 3 A flowchart illustrating the principle of model-based insertion learning index;

[0044] Figure 4 This is a schematic diagram of a learning index inserted based on the cache.

[0045] Figure 5 A flowchart illustrating the principle of model-based insertion learning index;

[0046] Figure 6 This is a schematic diagram of the waste recycling module;

[0047] Figure 7 This is a block diagram illustrating the principle of waste recycling.

[0048] Figure 8 This is a schematic diagram of the dynamic index table module;

[0049] Figure 9 This is a block diagram illustrating the principle of a dynamic index table. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention designs a method for accelerating read operations in a key-value storage system using learned indexes, aiming to improve the system's read performance on high-speed storage devices. First, the system's index granularity is changed from index blocks to index key-value pairs (KV-pairs), thus avoiding the need to search Bloom filters during index lookup. Then, all index information is constructed into an ordered index table. By constructing a learned index on the index table, the system can quickly predict the position of the target key in the index table, significantly improving the efficiency of index lookup during data reading. Given that a large number of key-value pairs may lead to excessive training overhead for the learned index, this invention adopts a key range-based partitioning strategy. Each partition (bucket) builds its own learned index model, significantly reducing the amount of data that needs to be processed for each learned index. Furthermore, this invention also constructs a learned index model for bucket location.

[0052] Since this invention uses index key-value pairs as the index granularity, it eliminates the need for compaction operations to sort data on disk. However, to avoid excessive space usage, we designed two garbage collection strategies: one is based on the proportion of garbage in each bucket, merging data within a bucket when the garbage reaches a certain threshold; the other is based on the amount of data in a bucket, adjusting the data volume by splitting the bucket, a process that also reclaims garbage data. Considering that the index table containing each key-value pair could be very large, directly storing it in memory would cause significant memory overhead. Therefore, this invention stores the index table on disk and dynamically loads active index table content into memory using a global translation dictionary, thus implementing a dynamic index table. This design maintains low memory footprint while achieving performance similar to a fully in-memory index table.

[0053] A system that uses a learned index to accelerate read operations in a key-value storage system includes a key space partitioning module, a learned index module, a data writing module, a data organization module, an index table update module, an index training module, a garbage collection module, a data retrieval module, a data reading module, and a dynamic index table module.

[0054] like Figure 1The key space partitioning module divides the entire key space into multiple consecutive intervals—buckets, each responsible for storing key-value pairs within a certain range. The learning index module locates buckets and learns and adjusts the position information of key-value pairs. The data writing module writes data into the hash buffer of the target bucket determined by the learning index. The data organization module sorts the data in the hash buffer after the data reaches a certain amount and writes it to the disk in the form of an ordered string table. The index table update module dynamically updates the learning index. The index training module retrains the learning index within the bucket after the index table is updated to ensure that it accurately reflects the latest distribution of key-value pairs. The garbage collection module performs intelligent garbage collection to maintain storage efficiency and system performance. The data retrieval module retrieves data from the learning index in the target bucket. The data reading module reads the required data from the disk based on the position information provided by the index after retrieving the data. The index table is stored on the disk, and the dynamic index table module dynamically loads active index content into memory through a global translation dictionary. This method provides performance comparable to a fully in-memory index table while maintaining low memory usage.

[0055] like Figure 2 and Figure 3 The learning index module includes a positioning module and an adjustment module; the positioning module is a learning index module based on model insertion, and the adjustment module is a learning index module based on cache insertion.

[0056] In the positioning module, the model insertion operation is based on the current distribution pattern of the buckets. Specifically, node i in the model represents the node at position i, which is responsible for indexing all key pairs whose calculated results fall within the interval [i, i+1). In the model, nodes are divided into two states: unused nodes and currently used nodes. Unused nodes point to the previous currently used node, while currently used nodes indicate the range of all key-value pairs from the current node to the next currently used node.

[0057] When the bucket undergoes structural changes, such as splitting or merging, the model of the present invention performs insertion or deletion operations accordingly; these operations require special care to ensure that all unused nodes between the inserted or deleted node and the next node in use correctly point to the new node in use.

[0058] To resolve potential conflicts when inserting nodes into the model, a local scaling mechanism is employed. Under this mechanism, any conflicting node is expanded into multiple nodes to maintain the model's accuracy and efficiency.

[0059] like Figure 4 and Figure 5The adjustment module is used to obtain the position information of key-value pairs (kv-pairs). In order to process newly inserted data, a hash table is used for temporary buffering. Once the amount of data in the hash table exceeds the set threshold, the system will automatically trigger the retraining of the model.

[0060] In designing the model, special attention was paid to ensuring low overhead in the learning and lookup processes, while also pursuing space efficiency. Piecewise Linear Regression (PLR) was chosen as the data modeling tool due to its simplicity and efficiency, meeting the requirements. PLR effectively constructs a model with error bounds by representing the sorted dataset as multiple line segments. In this model, the predicted location of each data point is guaranteed to be within an interval centered on the actual location and within a fixed error range. To construct this PLR model, we adopted an algorithm called Greedy-PLR. This algorithm processes data points one by one; if the current line segment cannot accommodate the new data point within the error bound, a new line segment is created. This method ensures that the construction process of the PLR ​​model is linear in time, i.e., linearly related to the number of data points. Once the model is trained, the lookup operation is very fast. First, a binary search is used to identify the line segment containing a specific key, and then the location of the key is predicted through simple linear operations. If the key is not in the predicted location, a fast local search is performed within a small range defined by the error bounds. The time complexity of the lookup operation is logarithmic and depends on the number of line segments. The space cost of the PLR ​​model is very low, because each line segment only requires a few dozen bytes of storage.

[0061] like Figure 6 and Figure 7 The garbage collection module is used to optimize the space efficiency of the data storage system. Its core objective is to reduce space amplification, that is, to avoid invalid data occupying valuable storage resources. This module implements two garbage collection mechanisms: a merge operation garbage collection mechanism and a bucket splitting garbage collection mechanism. Both of these mechanisms ensure high system performance and optimal data organization through a refined operation process.

[0062] The merge operation recycling mechanism begins to function after the index information is updated in the index table. The system first recalculates the proportion of invalid data within the buckets. When this proportion exceeds a preset threshold, a garbage collection process is triggered. Specific operations include: ST1, Data Sorting: All ordered string tables in the buckets are loaded into memory and re-sorted; ST2, Data Cleaning: Invalid garbage data is marked and removed during the sorting process; ST3, Data Rewriting: The cleaned valid data is rewritten back to disk; ST4, Index Update: Accordingly, the new position information of the key-value pairs within the buckets needs to be updated in the index table. This process does not require retraining the learned index model because the keys and their positions in the index table remain unchanged; only the physical storage location of the key-value pairs changes.

[0063] The bucket splitting operation is performed after the ordered string table is written to disk. The system evaluates whether the ratio of the amount of data in the bucket to the average level exceeds a certain threshold. If it does, the splitting process is initiated as follows: ST1', Data sorting and splitting: Similarly, all ordered string tables in the bucket are first loaded into memory and sorted, and then the data is equally split into two parts; ST2', Data allocation: The first part of the data is put back into the original bucket, while the second part is allocated to a newly created bucket; ST3', Index inheritance: The new bucket adopts the index information of the corresponding part in the original bucket; ST4', Model adjustment: The learned index models of the two buckets do not need to be completely retrained. Since the relative positions of the key-value pairs in the index table have not changed, only a few parameters in the model need to be adjusted; ST5', Index insertion: The newly generated bucket is inserted as a node into the model-based insertion learned index used to locate the bucket.

[0064] These two mechanisms work together to ensure that the system maintains efficient space utilization under dynamically changing data loads, while reducing the impact on performance by avoiding frequent and comprehensive retraining of the indexing model.

[0065] like Figure 8 and Figure 9The dynamic indexing module is designed to create a dynamic index table that achieves low memory consumption while maintaining high performance close to a full-memory index table. The system's index table is primarily stored on disk to minimize memory usage. In memory, a smaller cached index table and a global translation dictionary are maintained, both contributing to improved data retrieval efficiency. When the learned index model initiates a query request, the system first checks if the cached index table contains the required mapping information. If the relevant information is available in the cache, the system directly provides the required data from the cache, thus accelerating the reading process. If the requested mapping information is not found in the cached index table, it needs to be retrieved from the index table on disk and loaded into the cache. This process relies on the global translation dictionary, which guides the transfer of information from disk to cache. The global translation dictionary ensures the accuracy and consistency of mapping information transfer from disk to cache.

[0066] Considering the limited size of the cache index table, a segmented Least Recently Used (LRU) replacement algorithm is used to manage cached content and optimize the utilization of the cache index table. When the cache is full, the algorithm determines which entries should be evicted to make room for new mapping information. Since the index table and global translation dictionary are processed first in each write operation, the mapping information in the cache can be easily erased without complex additional operations.

[0067] like Figure 1 A method for accelerating read operations in a key-value storage system using learned indexes, comprising the following steps:

[0068] S1 divides the entire key space into multiple consecutive intervals, which are called "buckets". Each bucket is responsible for storing key-value pairs within a certain range. The size and range of the buckets are designed based on the data distribution and access patterns to maintain a uniform data distribution and optimal retrieval efficiency.

[0069] S2 employs a model-inserted learning index to locate buckets, making bucket location more efficient and accurate after ignoring minor overhead. The learning index within each bucket uses a buffered insertion technique, enabling efficient learning and adjustment of key-value pair position information. This learning index uses PLR to model the data, exhibiting low overhead during learning and lookup; the buffer structure used in the learning index is a hash table for temporarily storing data being written, optimizing write performance; and on disk, each bucket maintains multiple ordered string tables for persistent storage of sorted data.

[0070] S3 determines the bucket where the data should reside, writes and organizes the data, and updates the index table. Specifically, the data writing and organization process involves: when new data arrives, firstly, the learned index is used to determine the bucket where the data should reside, and the data is written to the hash buffer of the corresponding bucket; after a certain amount of data is accumulated, the contents of the hash buffer are sorted and written to disk as an ordered string table, while the index table is updated to reflect the new key-value pair positions.

[0071] S4 dynamically updates and trains the learning index. That is, after the index table is updated, the system will retrain the learning index in the bucket to ensure that it can accurately reflect the latest distribution of key-value pairs.

[0072] S5 performs intelligent garbage collection to maintain storage efficiency and system performance. Specifically, the garbage collection mechanism is divided into two types. The first is a merging operation, which is performed when the proportion of invalid data in a bucket reaches a certain level. This process clears the invalid data and updates the index table to maintain efficient use of storage space. The second is a splitting operation, which automatically performs a splitting operation to create new buckets to distribute the data in the face of continuous growth in the amount of data in a bucket. This ensures the performance of retrieval and write operations.

[0073] S6 performs data reading and retrieval. The specific method is as follows: when reading data, the target bucket is quickly located by relying on the learning index. After the bucket is determined, data retrieval is performed in the learning index. After the data is retrieved, the required data is read from the disk according to the location information provided by the index, ensuring fast and accurate data access.

[0074] S7 constructs a dynamic index table. The specific method is as follows: adopting an index table structure with disk storage as the main method to minimize memory usage, supplemented by a small cache index table in memory and a global translation dictionary to significantly improve data retrieval speed; and high-frequency data is accessed quickly in memory, while less frequently used data is kept on disk, achieving an optimized balance between efficient resource utilization and performance.

[0075] By combining traditional data organization methods with learned indexes, this invention not only improves the speed and accuracy of data retrieval but also maintains the high efficiency and long-term stability of the storage system through the introduction of an innovative garbage collection mechanism. This system design, which combines machine learning and data structure optimization, represents a significant advancement in data storage technology and is of great importance for processing large-scale datasets, especially in applications requiring fast read and write operations.

[0076] The core idea of ​​the aforementioned learning indexing mechanism is to use the distribution characteristics of data to train a prediction model that can predict the location or range corresponding to a given key value, thereby achieving efficient data retrieval.

[0077] The insertion operation of the learning index is divided into the following two types:

[0078] 1. Model-based Insertion: This type of insertion uses a trained machine learning model to predict the location of new data. When a new key-value pair is inserted, the model, which has learned the index, can predict which range of locations the data should be stored in. This method is particularly suitable for scenarios where the data distribution is relatively stable, in which case the model can accurately predict the insertion location, thereby significantly improving the efficiency of the insertion operation.

[0079] 2. Buffer-based Insertion: To handle situations where model predictions are insufficient or data distribution changes, the system provides a buffer-based insertion mechanism. In this mechanism, newly inserted data is first stored in a buffer. When the buffer reaches a certain threshold, the system triggers a batch insertion operation, potentially updating or retraining the prediction model to adapt to the new data distribution. This approach helps reduce insertion costs caused by model prediction errors and allows the system to adapt to changes in data distribution.

[0080] Specifically, such as Figure 2 and 3 This is a learned index that employs model insertion, primarily aimed at accurately and efficiently locating buckets. In the system proposed in this invention, the number of buckets is relatively small (e.g., several thousand), so space overhead is not a problem. Against this backdrop, this invention designs a pre-defined model whose insertion operation is based on the current distribution pattern of the buckets. A significant advantage of this method is that during the lookup operation, the model's computational result directly points to the target node, thereby eliminating error interference and improving lookup accuracy.

[0081] like Figure 4 and 5 This is a learning index that uses buffered insertion to efficiently retrieve the location information of key-value pairs (kv-pairs). A hash table is used for temporary buffering to handle newly inserted data. Once the amount of data in the hash table exceeds a set threshold, the system automatically triggers model retraining.

[0082] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the various illustrative components, blocks, modules, circuits, and steps are described above in a generalized manner in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the invention.

[0083] The prior description of this disclosure is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to this disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0084] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A system that uses a learned index to accelerate read operations in a key-value storage system, characterized in that: It includes a key space partitioning module, a learning index module, a data writing module, a data organization module, an index table update module, an index training module, a garbage collection module, a data retrieval module, a data reading module, and a dynamic index table module; The key space partitioning module divides the entire key space into multiple consecutive intervals—buckets—each responsible for storing key-value pairs within a certain range. The learning index module locates buckets and learns and adjusts the position information of key-value pairs. The data writing module writes data into the hash buffer of the target bucket determined by the learning index. The data organization module sorts the data in the hash buffer after the data reaches a certain amount and writes it to disk in the form of an ordered string table. The index table update module dynamically updates the learning index. The index training module retrains the learning index within the bucket after the index table is updated to ensure it accurately reflects the latest distribution of key-value pairs. The garbage collection module performs intelligent garbage collection to maintain storage efficiency and system performance. The data retrieval module retrieves data from the learning index in the target bucket. The data reading module reads the required data from the disk based on the position information provided by the index after retrieving the data. The index table is stored on the disk, and the dynamic index table module dynamically loads active index content into memory through a global translation dictionary. This method provides performance comparable to a fully in-memory index table while maintaining low memory usage.

2. The system according to claim 1, characterized in that: The learning index module includes a positioning module and an adjustment module; the positioning module is a learning index module based on model insertion, and the adjustment module is a learning index module based on cache insertion.

3. The system according to claim 2, characterized in that: In the positioning module, the model insertion operation is based on the current distribution pattern of the bucket; specifically, node i in the model represents the node at position i, which is responsible for indexing all key keys whose model calculation results fall within the interval [i, i+1); in the model, nodes are divided into two states: unused nodes and nodes in use; unused nodes point to the previous node in use, while nodes in use indicate the range of all key-value pairs from the current node to the next node in use. When the bucket undergoes structural changes, such as splitting or merging, the model of the present invention performs insertion or deletion operations accordingly; these operations require special care to ensure that all unused nodes between the inserted or deleted node and the next node in use correctly point to the new node in use.

4. The system according to claim 3, characterized in that: To resolve potential conflicts when inserting nodes into the model, a local scaling mechanism is employed. Under this mechanism, any conflicting node is expanded into multiple nodes to maintain the model's accuracy and efficiency.

5. The system according to claim 2, characterized in that: The adjustment module is used to obtain the position information of key-value pairs (kv-pairs). To process newly inserted data, a hash table is used for temporary buffering. Once the amount of data in the hash table exceeds the set threshold, the system will automatically trigger the retraining of the model.

6. The system according to claim 1, characterized in that: The waste recycling module implements two waste recycling mechanisms: a combined recycling mechanism and a bin-splitting recycling mechanism.

7. The system according to claim 6, characterized in that: The merge operation recycling mechanism starts working after the index information is updated to the index table; the system first recalculates the proportion of invalid data in the bucket, and when this proportion exceeds a preset threshold, the garbage collection process will be triggered. The specific operations include: ST1, Data Sorting: All ordered strings in the buckets will be loaded into memory and re-sorted; ST2, Data Cleaning: During the sorting process, invalid and garbage data will be marked and removed; ST3, Data Rewrite: The cleaned and valid data is rewritten back to the disk; ST4, Index Update: Accordingly, the new position information of the key-value pairs in the bucket needs to be updated in the index table.

8. The system according to claim 6, characterized in that: Bucket splitting and recycling are performed after the ordered string table has been written to disk. The system evaluates whether the ratio of the amount of data in the bucket to the average level exceeds a certain threshold. If it does, the splitting process is initiated as follows: ST1', Data Sorting and Splitting: Similarly, all ordered string tables in the bucket are first loaded into memory and sorted, and then the data is equally split into two parts; ST2', Data Allocation: The first part of the data will be put back into the original bucket, while the second part will be allocated to a newly created bucket; ST3', Index Inheritance: The new bucket uses the corresponding part of the index information in the original bucket; ST4', Model Adjustment: The two-bucket learning index model does not need to be completely retrained. Since the relative positions of key-value pairs in the index table have not changed, only a few parameters in the model need to be adjusted. ST5', Index Insertion: The newly generated bucket will be inserted as a node into the model-based insertion learning index used to locate the bucket.

9. The system according to claim 1, characterized in that: The dynamic indexing module is designed to create a dynamic index table that achieves low memory consumption while maintaining high performance close to that of a full-memory index table. The system's index table is primarily stored on disk to minimize memory usage. In memory, a cache index table and a global translation dictionary are maintained. When the learning index model initiates a query request, the system first checks whether the cache index table contains the required mapping information. If the relevant information is available in the cache, the system will directly provide the required data from the cache, thereby speeding up the reading process; if the requested mapping information is not found in the cache index table, it needs to be retrieved from the index table on disk and loaded into the cache.

10. The system according to claim 9, characterized in that: Considering the limited size of the cache index table, a segmented Least Recently Used (LRU) replacement algorithm is used to manage cached content and optimize the utilization of the cache index table. When the cache is full, the algorithm determines which entries should be evicted to make room for new mapping information. Since the index table and global translation dictionary are processed first in each write operation, the mapping information in the cache can be easily erased without complex additional operations.

Images