Method and system for efficient learning index in log-structured merge tree kv store

Abstract

A method and system for dual target learning index in a key-value store is disclosed. The key-value store is a log-structured merge tree based storage. A sorted string table is divided into a plurality of data blocks, wherein a sum of values of keys in each data block is less than or equal to a maximum block size value. Piecewise linear approximation functions are generated for the plurality of data blocks, wherein a maximum interpolation error of the piecewise linear approximation functions is less than or equal to a model error value. The maximum block size value and the model error value are provided by a reinforcement learning agent. Upon receiving a lookup query request to find a target key, the system initiates a discovery operation to discover a data block from the plurality of data blocks and generates a search range of the discovered data block.

Description

Technical Field

[0001] The present invention generally relates to methods and systems for key-value storage, and more particularly to systems, devices, and methods for supporting bi-objective learning indexes for storage based on log structure merge trees. Background Technology

[0002] This innovation targets a learning indexing system built upon the log-structured merge-tree (LSM) tree of a key-value (KV) database. A KV database or KV store is a data storage paradigm designed for storing, retrieving, and managing associative arrays; it's also a data structure, now more commonly known as a dictionary or hash table. A dictionary can contain a collection of objects or records, each with many distinct fields, each containing data. These records are stored and retrieved using keys that uniquely identify them and are used to locate data within the database. An LSM tree is a disk-based data structure widely used in storage systems and databases to efficiently handle high-write workloads. LSM trees are optimized for write-intensive workloads while still providing efficient read operations. Learning indexes need to satisfy certain data attributes of the original data. Data management systems can be modeled as a series of mathematical operations. Machine learning (ML) can be applied to indexing data and creating ML-based data structures.

[0003] Data structures in memory can be used as write buffers and can be called memory tables. Memory tables store recently written data in sorted order by key. Write operations are first appended to the memory table for efficient sequential writes in memory. Once the memory table reaches a certain size or threshold, it can be flushed to disk as a sorted string table (SST). An SST is an immutable, sorted table stored on disk. SSTs are typically stored in multiple files. Each file represents a separate level in an LSM tree. SSTs are optimized for sequential reads. An SST is divided into multiple data blocks. This means that the data in an SST can be stored in the same location within the same region, depending on the size of the key-value pairs stored in the system.

[0004] Learning indexes aim to improve database retrieval efficiency by mapping keys to locations using machine learning models. These models can range from neural networks to simpler hierarchical models. Learning indexes have historically used ensemble learning and hierarchical structuring. Complex neural network-based models can require complex training and inference. On the other hand, piecewise linear models used to approximate the cumulative distribution function (CDF) for key prediction may be more memory-efficient than traditional indexes. Despite the potential of learning indexes, recent research suggests that in persistent storage environments, their performance may not outperform traditional methods, such as B+ tree-based data storage. The main issue is that learning indexes may not effectively utilize disk storage characteristics, leading to multiple disk input and output (I / O) operations during the final search phase. This problem also affects learning indexing techniques applied to LSM trees.

[0005] Learning indexing techniques primarily focus on fixed-size keys, with limited attention paid to variable-length string keys. Radix String Spline (RSS) methods have made significant progress through trie-based search. (A trie-based data structure was designed to quickly retrieve objects via string search.) RSS combines keys into manageable segments and computes a piecewise linear CDF model. However, RSS still suffers from inefficiency in the final search phase, where most of the time is spent, highlighting the need for further optimization in learning indexes for string keys.

[0006] The application of the LSM tree storage engine in three different use cases at Meta—structured query language (SQL) databases, distributed key-value stores, and artificial intelligence (AI) / ML services—demonstrates that keys and values ​​can have variable sizes, with small standard deviations for keys and larger deviations for values. However, the LSM tree learning indexing scheme only supports fixed-size keys, making it impractical for these use cases.

[0007] RocksDB is a high-performance, embeddable key-value store approach that can be used as a storage engine, as described by Dong, S. et al. in "Evolution of development priorities in key-value stores serving large-scale applications: The RocksDB experience" (19th USENIX File and Storage Technology Conference (FAST21), February 2021, USENIX Association, pp. 33–49). Some systems utilize RocksDB's default indexing mechanism. RocksDB uses LSM trees for indexing, a combination of techniques that proportionally distribute the index load. It uses Bloom filters and data partitioning. This method is called TableIndex in RocksDB.

[0008] LeaderKV is a read-optimized LSM tree-based key-value store, as disclosed by WANG, Y. et al. in "LeaderKV: Improving read performance of KV stores via learned index and decoupled KV table" (40th IEEE International Conference on Data Engineering, ICDE 2024, May 13-16, pp. 29-41). LeaderKV employs decoupled key-value tables (DK tables) and efficient learned indexes for data retrieval. This approach separates keys and values ​​and improves read performance by removing value size differences from the learned index model by storing values ​​separately. Figure 1 A table is shown that summarizes the properties of LeaderKV and other LSM tree-based KV stores.

[0009] WiscKey is a persistent key-value store based on an LSM tree with a performance-oriented data layout that separates keys and values ​​to minimize I / O amplification. Bourbon is another key-value store that employs a learned index, optimizing lookups by understanding key distribution. Based on Wisckey (a variant of LevelDB), Bourbon maintains a fixed block size but may require loading multiple blocks, as disclosed by DAI, Y. et al. in "From Wisckey to Bourbon: a learned index for log-structured merge trees" (Proceedings of the 14th USENIX Operating System Design and Implementation Conference, OSDI 2020, USENIX Association, pp. 155–171).

[0010] TridentKV is a learning index version of RocksDB designed to retrieve key-value pairs by loading only a single data block, as disclosed by LU, K. et al. in "TridentKV: A read-optimized lsm-tree based kv store via adaptive indexing and space-efficient partitioning" (IEEE Transactions on Parallel and Distributed Systems 33, 8 (2022), pp. 1953–1966). TridentKV modifies the data block size, which can sometimes result in significantly larger blocks, impacting lookup performance when accessing large blocks. Open-source code for TridentKV is available, built upon RocksDB.

[0011] CN117389950A, published on January 12, 2024, discloses a method for adding a learning index to each layer of an LSM and using a recursive model index (RMI) to improve the efficiency of file segment lookup.

[0012] CN116382588A, published on July 4, 2023, discloses a method for separating data based on the distribution of data stored in an SST file. Addressing the complexity of this data distribution, this design uses a recursive method (e.g., RMI) and buffered data to arrange the data for a recursive model. Therefore, this method can handle dynamic insertions with better performance.

[0013] While learned indexes in known schemes improve indexing efficiency, they often neglect the data access phase, leading to high latency during block retrieval. This high latency is due to multiple block read operations performed during the data access phase of the learned index. Many schemes do not consider the need to optimize both the indexing and data access phases simultaneously. Most known schemes use a fixed block size, which can result in inaccurate block predictions or blocks exceeding the required size, increasing I / O time and read latency. The final step of the lookup process, retrieving the actual key-value pairs from the block, is typically inefficient. Many schemes fail to optimize this phase, resulting in additional latency that negates the benefits of learned indexes.

[0014] Existing solutions are mainly designed for fixed-size keys, which limits their applicability in real-world scenarios where keys and values ​​have variable lengths.

[0015] Therefore, there is a need to improve the methods and systems for efficient learning indexing in log-structured merged KV stores. Summary of the Invention

[0016] This invention relates to methods, apparatus, and systems for efficient learning indexing in a log-structured merged-tree key-value store. According to embodiments of the invention, a method for bi-objective learning indexing in a log-structured merged-tree key store is provided. The method is performed by an electronic device. The method includes: providing a plurality of keys; generating a sorted string table (SST) by arranging the key values ​​of the plurality of keys in ascending order; and indexing the corresponding keys of the SST (using a counting sequence of positive integers). Upon receiving a request, the method further includes: requesting and receiving a maximum block size value (Bmax) and a model error value (E) from a reinforcement learning (RL) agent; dividing the SST into a plurality of data blocks (wherein each data block includes one or more corresponding keys); generating a piecewise linear approximation (PLA) function; and storing the plurality of data blocks, the PLA function, and the E. In some embodiments of the method, the sum of the values ​​of the one or more corresponding keys of each data block is less than or equal to the Bmax. In other embodiments of the method, the first key of the first data block is the first spline point of the first data block, and the last key of each data block is the spline point of each data block. The PLA function is constructed based on the spline points of each data block and the first spline point of the first data block. The maximum interpolation error of the PLA function is less than or equal to E.

[0017] In some embodiments, the method may include: receiving an indication of a target key, wherein the indication is at least partially based on a pointer, implemented data, or a combination thereof; receiving a lookup query request to find the target key among the plurality of keys; and determining whether the value of the target key is within the key value range of the SST. If the value of the target key is within the key value range of the SST, a discovery operation is initiated to discover a data block from the plurality of data blocks. The method may further include: inputting the value of the target key into a function and running the function to generate a value (V) at the center of a search range, generating the search range of the data block, initiating a discovery operation to discover the target key within the search range, and determining whether the target key has been found within the search range. If the target key has been found within the search range, the target key is reported along with a corresponding pointer to a value block.

[0018] In some embodiments of the method, the function may be the PLA function or a piecewise regression approximation (PRA) function. In other embodiments, the data block has a corresponding key value range that overlaps with the value of the target key, and / or the search range of the data block is defined between (V – E) and (V + E). In some embodiments of the disclosed method, the plurality of data blocks, the PLA function, and E are stored in persistent storage. In some embodiments, the disclosed method may further include: retrieving a simplified index model from persistent storage before initiating a discovery operation to discover the data block from the plurality of data blocks, and / or retrieving the data block from persistent storage after the discovery operation for discovering the data block from the plurality of data blocks. In some embodiments of the method, the discovery operation for discovering the target key is a binary search or a linear search. Other embodiments may further include: requesting and receiving an indication of the most significant bit from the RL agent before initiating the discovery operation to discover the target key. In some embodiments, the discovery operation for discovering the target key may include: selecting an instant key within the search range and comparing each bit of a portion of the value of the instant key with a corresponding bit of the value of the target key, wherein the portion of the value of the instant key is defined by the first bit and the most significant bit. Some embodiments of the method may include: the RL agent recording the latency of the lookup query request and adjusting parameters for future lookup query requests.

[0019] Some embodiments disclosed in this application pertain to a storage system. The storage system is a log-structured merge tree-based storage system, and the disclosed storage system includes an electronic device. The electronic device is configured to: provide multiple keys; generate a sorted string table (SST) by arranging the key values ​​of the multiple keys in ascending order; index the corresponding keys of the SST using a sequence of positive integers; receive a request; request and receive a maximum block size value (Bmax) and a model error value (E) from a reinforcement learning (RL) agent; divide the SST into multiple data blocks (wherein each data block includes one or more corresponding keys); generate a piecewise linear approximation (PLA) function; and store the multiple data blocks, the PLA function, and the E.

[0020] In some embodiments of the storage system, the electronic device is further configured to: receive an indication of a target key, wherein the indication is at least partially based on a pointer, implemented data, or a combination thereof; receive a lookup query request to find the target key among the plurality of keys; determine whether the value of the target key is within the key value range of the SST; if the value of the target key is within the key value range of the SST, initiate a discovery operation to discover a data block from the plurality of data blocks; input the value of the target key into a function and run the function to generate a value (V) at the center of the search range; generate the search range of the data block; initiate a discovery operation to find the target key within the search range; determine whether the target key has been found within the search range; if the target key has been found within the search range, report the target key along with a corresponding pointer to the value block. In some embodiments of the system, the function is the PLA function or a piecewise regression approximation (PRA) function. In other embodiments, the sum of the values ​​of the one or more corresponding keys of each data block is less than or equal to Bmax. In some embodiments of the disclosed system, the first key of the first data block is the first spline point of the first data block, the last key of each data block is the spline point of each data block, the PLA function is constructed based on the spline points of each data block and the first spline point of the first data block, and the maximum interpolation error of the PLA function is less than or equal to E. In other embodiments of the system, the search range of the data block is defined between (V – E) and (V + E).

[0021] According to an embodiment, the electronic device in a communication network includes a processor, a network interface, and a memory, and is used to perform one or more methods described herein. According to an embodiment, a system of such electronic devices is provided, which are networked together and used to interact to perform one or more methods described herein.

[0022] According to embodiments of the present invention, a computer program product is provided, the computer program product comprising a (e.g., non-transitory) computer-readable medium storing statements and instructions, wherein the statements and instructions, when executed by one or more computer processors, cause the computer processors to perform the methods described above. The computer processor may be part of one or more electronic devices (e.g., network entities) described herein.

[0023] Embodiments have been described above in conjunction with various aspects of the present invention, and these embodiments can be implemented based on these aspects. Those skilled in the art will understand that embodiments can be implemented in conjunction with the aspects describing these embodiments, but may also be implemented together with other embodiments of that aspect. It will be apparent to those skilled in the art that embodiments are mutually exclusive or inconsistent with each other. Some embodiments may be described in conjunction with one aspect, but may also be applicable to other aspects, as will be apparent to those skilled in the art.

[0024] All implementations of the present invention have at least one of the above-described aspects, but not necessarily all of them. It should be understood that some aspects of the present invention are intended to achieve the above-described objectives, but may not satisfy those objectives and / or may satisfy other objectives not specifically described herein.

[0025] Additional and / or alternative features, aspects, and advantages of the present invention will be apparent from the following description, the accompanying drawings, and the appended claims. Attached Figure Description

[0026] To better understand the present invention and its other aspects and further features, reference is made to the following description used in conjunction with the accompanying drawings, in which: Figure 1 A table is shown that summarizes the attributes of the LSM tree-based KV store.

[0027] Figure 2 A representative example of SST is shown.

[0028] Figure 3A Representative examples of DobLIX in an LSM tree-based KV store provided by various non-limiting embodiments of the present invention are shown.

[0029] Figure 3B A flowchart illustrating a representative example of the DobLIX method is shown.

[0030] Figure 4A Representative examples of dual-target PLAs provided by various non-limiting embodiments of the present invention are shown.

[0031] Figure 4B A flowchart illustrating a representative example of a PLA is shown.

[0032] Figure 5 Representative examples of the PLA function Pr provided by various non-limiting embodiments of the present invention are shown.

[0033] Figure 6A Representative examples of the search process (last-mile search) provided by various non-limiting embodiments of the present invention are shown.

[0034] Figure 6B A flowchart illustrating a representative example of the search process (last-mile search) is shown.

[0035] Figure 7A A flowchart illustrating a representative example of DobLIX provided in certain non-limiting embodiments is shown.

[0036] Figure 7B A flowchart is shown, illustrating representative examples provided by certain non-limiting embodiments.

[0037] Figure 8 An electronic device for implementing the method operation provided in the embodiments of the present invention is shown. Detailed Implementation

[0038] There are two main approaches to reduce the complexity of big data management. One approach is data partitioning (Par(.)): dividing a large dataset D into smaller partitions. The other approach is indexing (I(.)): mapping query keys directly to the target data in storage. To construct a comprehensive data pipeline for dataset D, a common strategy is to combine P={Par1,Par2,…,Par…} in a specific order. n} and I={I1,I2,…,I m The sequence is represented as F = f1(f2(…f…). k (D)))=f1○f2○…○f k (D). Here, f i Belongs to partition P, f i The index i is in the range [1:n]. i The index i is in the range [1:m]. Within the context of an LSM tree, the data is structured into four distinct partitions, defined as P. LSM ={Par TreeLevel Par SST ParBlock Par KV The corresponding indexing mechanism is provided by I. LSM ={I LevelBloomFilter ,I SST ,I BlockIndex The composite function is represented by F. LSM =Par KV ○I BlockIndex ○Par Block ○Par SST ○I SST ○Par TreeLevel ○I LevelBloomFilter (D). Decomposition of F LSM First, the given data D is processed through a tree hierarchy (Par TreeLevel The partition is divided into a hierarchical structure and uses a hierarchical Bloom filter (I). LevelBloomFilter The hierarchical Bloom filter generates Boolean indexes, which indicate whether a key exists at a specific level. Within each level, data can be divided into different SST files (Par...). SST Each SST maintains a key-range index, which is used as the SST index (I SST Each SST arranges the data in a sorted manner and further divides it into key-value pairs (Par). KV ) unified block (Par Block Additionally, each SST includes an index block in its SST metadata, which contains index entries for each data block (I). BlockIndex ).

[0039] Improving LSM tree search performance through integrated learned indexes is one aspect of this invention. Learned index schemes tailored to LSM structures significantly improve index retrieval efficiency. However, a significant limitation of these works may lie in data access overhead, particularly involving retrieving blocks from storage to memory. Figure 2 SST(A) includes blocks B1, B2, B3, B4, and B5 of equal size, and SST(B) includes blocks B1, B2, and B3 of equal size. SST(A) and SST(B) are optimized for read operations. However, adhering to a fixed block size may lead to inaccurate block lookups and require accessing multiple blocks during the read operation. Figure 2The diagram also illustrates an SST(C) including blocks of variable size B1, B2, B3, B4, and B5. This can result in significantly larger blocks, increasing block I / O time. This issue exists with SST(C). These dilemmas stem from isolated optimizations of the learned index model without considering the data access components. While customizing the SST block size can reduce average latency compared to LSM tree indexes, this can also lead to increased tail latency in scenarios requiring the retrieval of multiple or large blocks. Figure 2 The SST(D) also includes blocks B1 through B5. The size of each block in the SST(D) is less than or equal to Bmax. Applying the double-objective learned index (DobLIX) method for LSM-based KV storage disclosed in this application guarantees that only one block of the SST(D) is read, and the block size does not exceed the maximum block size Bmax.

[0040] DobLIX optimizes both the indexing and data access phases. DobLIX reduces read latency by minimizing block retrieval. This method employs dictionary optimization to ensure the learned index model not only accurately predicts key locations but also minimizes block retrieval. (Dictionary optimization in multi-objective systems aims to find the optimal value, decomposing the problem into subproblems with a single objective and treating the remaining parameters as non-negotiable.) This invention introduces a learned index designed to handle real-world workloads with variable-size keys using a Radix String Spline (RSS) model. This approach ensures that the index efficiently predicts blocks containing target key-value pairs, even for keys of varying lengths. DobLIX improves last-mile search efficiency by narrowing the search range within blocks and refining the final step of the lookup process by tuning the learned index model. Reinforcement learning (RL) agents dynamically adjust the parameters of the learned index (e.g., maximum permissible error and block size) to adapt to different workloads and datasets. DobLIX's core strengths in indexing and data access optimization ensure its long-term market adaptability and relevance.

[0041] The DobLIX architecture operates at the SST level, where DobLIX learns a model for each SST based on the distribution of key-value pairs. This model is then stored in the SST metadata so that it can be used during read operations to predict relevant blocks for a given key. DobLIX is suitable for scenarios where high read performance is critical. DobLIX is designed to integrate with LSM-tree-based KV stores such as RocksDB, LevelDB, or Cassandra to improve read performance. Cloud-based databases and storage systems can benefit from DobLIX's dual-objective optimization for faster data retrieval. Data-intensive AI and ML workflows that rely on fast data access from LSM-tree-based storage systems can leverage DobLIX to reduce latency.

[0042] The system architecture includes an LSM tree, which is the underlying storage structure where SSTs are organized hierarchically. The architecture also includes a learned indexing model: for each SST, DobLIX trains a model that predicts the blocks containing the target key based on the cumulative distribution function (CDF) of the keys. Some embodiments of the system may also include a Reinforcement Learning (RL) agent. The RL agent continuously monitors the performance of the learned index and tunes parameters to optimize indexing and data access. The RL agent can dynamically adjust the parameters of the learned index (e.g., maximum permissible error and block size) to adapt to different workloads and datasets, thereby improving performance over time.

[0043] Figure 3A Representative examples of the DobLIX method executed by a processor in an LSM tree-based KV storage provided by various non-limiting embodiments of the present invention are shown. Figure 3B A flowchart illustrating a representative example of the DobLIX method is shown.

[0044] At action 1, receive the target key (k) i The search query is initiated. Upon receiving the search query, a check operation is started to examine the active memory table. If the target key is not found, proceed to action 2. In action 2, a check operation is started to examine the target key k. i An immutable memory table. If the target key is not found, proceed to action 3.

[0045] At action 3, identify the SST containing the relevant range of the key. j–2 At action 4, SST j–2 The learned index (Pr) and model error value (E) are loaded into memory. At actions 5 through 7, the learned index Pr is used to predict the value containing the target key k. i Block Bt (Read only once) and generate index value M(k) i At action 8, retrieve block B from persistent storage. t Then load the block into memory. At action 9, in block B... t A binary search is performed to find the exact key-value pair. The search range is M(k). i )–E to M(k i )+E. At action 10, the RL agent records the query delay and adjusts system parameters for future queries.

[0046] The disclosed invention employs machine learning-based indexing to optimize the lookup process. This optimization falls under the category of dictionary optimization, while also improving the block indexing and block read I / O portions. Dictionary optimization provides a structured approach to solving multi-objective optimization problems by ranking and prioritizing objectives based on their importance. First, the highest priority objective is optimized; then subsequent objectives are processed while maintaining adherence to the feasible set of previous optimizations. This sequential approach in dictionary optimization ensures that each objective is optimized in order of importance without altering previously achieved results. By processing one objective at a time within a clear hierarchical framework, dictionary optimization aids in decision-making. Dictionary optimization can provide deterministic solutions for managing multiple objectives, which is particularly advantageous for fundamental objectives in fields such as resource allocation or scheduling. By ensuring that critical objectives are fully satisfied, dictionary optimization allows secondary objectives to be optimized within the remaining feasible solution space.

[0047] By learning the cumulative distribution of keys stored in the table, a learning index model can be trained to predict the position of keys in the SST. The learning process can use the Piecewise Linear Approximation (PLA) method. This method divides the key space into multiple blocks and uses linear splines to estimate the position of keys in each block. The generation of blocks and learning indexes ensures that: (1) the learning index allows reading only one block, and (2) the block size does not exceed the maximum block size. Figure 4A Representative examples of processor-executed dual-target PLA methods provided by various non-limiting embodiments of the present invention are shown.

[0048] Figure 4BA flowchart illustrating a representative example of PLA method 400 is shown. At action 401, the model error value (E) and the maximum block size value (Bmax) are received from the RL agent; a first key-value pair is added to the first block; a cardinality list is started, and the current index number is set to "0". At action 402, a check operation is initiated to check if the current block size is greater than the specified maximum block size. If the current block size is greater than the specified Bmax, the last key value in the current block is added to the spline points for indexing, and the current key is added to a new block. At action 403, a check operation is initiated using a function that calculates the maximum distance (maximum interpolation) between points in the set and the given line to check if the current spline reaches E. If the current spline reaches E, the last key value in the current block is added to the spline points for indexing, and the current key is added to a new block.

[0049] At action 404, if both the check operation for whether the current block size is greater than Bmax and the check operation for whether the current spline reaches E provide negative results, then the current key-value pair is added to the current block. At action 405, the next key-value pair is selected, and the process returns to action 402.

[0050] Figure 5 Representative examples of the PLA function Pr executed by the processor, provided by various non-limiting embodiments of the present invention, are shown. When the size of the added point exceeds Bmax, a new block is created to maintain the optimal I / O performance of the previous block, such as... Figure 5 The crosses in the diagram indicate this mechanism. This mechanism generates a new set of spline points, which, in addition to the standard block formed by E, also introduces new blocks when the quadratic optimization criterion is satisfied.

[0051] After learning the index model and retrieving the appropriate data block from storage, the system performs a last-mile search within the block to find the exact key-value pair. Figure 6A Representative examples of processor-executed lookup processes (last-mile search) provided by various non-limiting embodiments of the present invention are illustrated. The learned index provides close approximations of key positions, allowing the system to perform binary search within a small range of KV pairs. For string-based keys, the system compares only the significant portions of the key, thereby reducing the overhead of full-key comparisons.

[0052] Figure 6BA flowchart illustrating a representative example of the lookup process (last-mile search) 600 is shown. At action 601, the model trie is traversed: at each level, the level is captured and the appropriate child is found; at the leaf level, the location of the block is obtained and the block is pulled from storage. At action 602, E, the value of the target key, and the level L information are provided. At action 603, using the model output of E and the target key, the search range of the key entries required for the last-mile search is calculated. At action 604, using the level L information and the prefix size K, the offset (range start point) for string comparison is calculated, ignoring common prefixes in the block, and comparing only K bytes of the string starting from the offset.

[0053] Figure 7A A flowchart of a representative example 700A of DobLIX provided in some non-limiting embodiments is shown. At action 701, multiple keys are provided. At action 702, a sorted string table (SST) is generated by arranging the key values ​​of the multiple keys in ascending order. The SST can also be represented as a sorted sequence of keys (SSK). At action 703, the corresponding keys of the SST are indexed using a sequence of positive integers. At action 704, a request is received. After receiving the request, at action 705, a maximum block size value (Bmax) and a model error value (E) are requested from and received from the reinforcement learning (RL) agent. At action 706, the SST is divided into multiple data blocks, where each data block includes one or more corresponding keys. Data blocks can also be represented as a block sequence. At action 707, a piecewise linear approximation (PLA) function is generated. At action 708, the block sequence, the PLA function, and E are stored.

[0054] Figure 7B A flowchart of a representative example 700B provided in some non-limiting embodiments is shown. At action 709, the target key (k) is received. i The instructions are as follows: At action 710, a lookup query request to find the target key among multiple keys is received. At action 711, it is determined whether the value of the target key is within the key value range of the SST. At action 712, if the value of the target key is within the key value range of the SST, a discovery operation is initiated to discover a data block from multiple data blocks that has a corresponding key value range that overlaps with the value of the target key. At action 713, the value of the target key is input into the PLA function, and the PLA function is run to generate the index value (M(k...). iThe index value can be described as the value (V) of the center of the search range. At action 714, the search range for the data block is generated. At action 715, a discovery operation is initiated to find the target key within the search range. At action 716, it is determined whether the target key has been found within the search range. If the target key has been found within the search range, it is reported along with the corresponding pointer to the value block.

[0055] While the DobLIX embodiments described above are optimized for LSM-tree-based KV stores, other embodiments of the dual-objective learning indexing method can be extended to other storage architectures, such as B-trees or columnar databases. These systems can benefit from dual optimization in both the indexing and data access phases, thereby improving read performance in scenarios where data retrieval latency is a bottleneck. DobLIX efficiently handles variable-size key-value pairs and can be used to manage other types of data, such as graph databases or document-oriented databases. These databases store complex hierarchical data structures, and learning indexes adapted to different data distributions can significantly improve performance. In other embodiments, the RL agent used by DobLIX for fine-tuning parameters can be further developed to support more complex learning methods. By integrating more advanced models (e.g., deep learning-based predictors), corresponding versions of DobLIX can achieve higher accuracy in predicting key positions, further reducing block accesses and increasing throughput.

[0056] Distributed databases, especially those used in large-scale cloud environments, can integrate DobLIX to improve data retrieval performance across multiple nodes. For example, distributed key-value stores can benefit from DobLIX's reinforcement learning-based tuning agent, which dynamically adjusts parameters based on workload characteristics. This can optimize both local and distributed data access simultaneously, reducing latency in geographically distributed systems.

[0057] Figure 8A block diagram of an electronic device 800 for performing the methods of the present invention is shown. Device 800 can wirelessly communicate with one or more other devices. Device 800 may include a computer processor operatively coupled to computer memory. A computer with network capabilities, including a wireless transceiver, may be configured as device 800. Device 800 may correspond to a portion of: a computer server, or a network node providing network access (e.g., an IEEE 802.11 access point (AP) or similar device), or a network node accessing a network (e.g., an IEEE 802.11 wireless station (STA)). Device 800 can wirelessly couple to other devices via a WLAN, such as an IEEE 802.11 compliant wireless local area network (WLAN).

[0058] like Figure 8 As shown, device 800 includes a processor 801 (e.g., a dedicated processor such as a Central Processing Unit (CPU) or a Graphics Processing Unit (GPU), or other such processor units), memory 804, non-transient mass storage 802, I / O interface 805, network interface 803, and wireless transceiver 806, all of which are communicatively coupled via a bidirectional bus 807. The transceiver 806 includes one or more antennas. According to some embodiments, any or all of the elements may be utilized, or only a subset of the elements may be utilized. Furthermore, device 800 may contain multiple instances of certain elements, such as multiple processors, memories, or transceivers. Additionally, elements of the hardware device may be directly coupled to other elements without requiring a bidirectional bus. Alternatively or additionally, in addition to processors and memory, other processing electronics (e.g., application-specific integrated circuits, field-programmable gate arrays, digital circuits, or analog circuits, or combinations thereof) may be used to perform the required logical operations. Each integrated circuit may include one or more semiconductor chips and semiconductor chiplets.

[0059] Memory 804 may include any type of non-transitory memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or any combination thereof. Mass storage element 802 may include any type of non-transitory storage device, such as a solid-state drive, hard disk drive, disk drive, optical disk drive, USB flash drive, or any computer program product for storing data and machine-executable program code. According to some embodiments, memory 804 or mass storage 802 may record statements and instructions executable by processor 801 thereon for performing any of the above-described method operations.

[0060] It should be understood that although specific embodiments of the invention have been described herein for illustrative purposes, various modifications can be made without departing from the scope of the invention. Therefore, the specification and drawings are to be considered merely as a description of the invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of the invention. Specifically, providing computer program products or program elements for storing machine-readable signals, or program storage or memory devices such as magnetic wires, magnetic tapes, disks or optical fibers, optical tapes or optical discs, within the scope of the invention, for controlling the operation of a computer by methods according to the invention and / or constructing some or all of its components according to the invention.

[0061] The actions associated with the methods described herein can be implemented as coded instructions in a computer program product. In other words, a computer program product is a computer-readable medium on which software code is recorded to execute the methods when the computer program product is loaded into memory and executed on the microprocessor of a wireless communication device.

[0062] Furthermore, each operation of the method can be executed on any computing device (e.g., a personal computer, server, PDA) and is performed based on one or more program elements, modules, or objects, or a portion thereof, generated from any programming language (e.g., C++ or Java). Additionally, each operation, or the file or object implementing each operation, can be executed by dedicated hardware or a circuit module designed for this purpose.

[0063] Based on the description of the above embodiments, the present invention can be implemented solely in hardware, or it can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product. The software product can be stored in a non-volatile or non-transient storage medium, such as a compact disk read-only memory (CD-ROM), a USB flash drive, or a removable hard drive. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to perform the methods provided in the embodiments of the present invention. For example, such performance may correspond to the simulation of the logical operations described herein. According to embodiments of the present invention, the software product may additionally or alternatively include multiple instructions that enable a computer device to perform operations configuring or programming digital logic devices.

Claims

1. A method for bi-objective learning indexing in a key storage based on a log-structured merge tree, characterized in that, The method includes the electronic device performing the following operations: Provide multiple keys; A sorted string table (SST) is generated by arranging the key values ​​of the multiple keys in ascending order. The corresponding keys of the SST are indexed using a sequence of positive integers; Receive request; Request and receive the maximum block size value (Bmax) and model error value (E) from the reinforcement learning (RL) agent. The SST is divided into multiple data blocks, wherein each data block includes one or more corresponding keys; Generate a piecewise linear approximation (PLA) function; Store the plurality of data blocks, the PLA function, and the E.

2. The method according to claim 1, characterized in that, The sum of the values ​​of the one or more corresponding keys of each data block is less than or equal to Bmax.

3. The method according to claim 1, characterized in that, The first key of the first data block is the first instance of the first data block. The final key of each data block is the spline point of each data block. The PLA function is constructed based on the spline points of each data block and the first spline points of the first data block. The maximum interpolation error of the PLA function is less than or equal to E.

4. The method according to claim 1, characterized in that, Also includes: Receive an indication of a target key, wherein the indication is at least in part based on a pointer, implemented data, or a combination thereof; Receive a search query request to find the target key among the plurality of keys; Determine whether the value of the target key is within the range of key values ​​in the SST; If the value of the target key is within the range of the key values ​​of the SST, then a discovery operation is initiated to discover the data block from the plurality of data blocks; The value of the target key is input into the function, and the function is run to generate the value (V) of the center of the search range. The search range for generating the data block; Initiate a discovery operation to find the target key within the search range; Determine whether the target key has been found within the search range; If the target key is found within the search range, the target key is reported along with the corresponding pointer to the value block.

5. The method according to claim 4, characterized in that, The function is either the PLA function or the piecewise regression approximation (PRA) function.

6. The method according to claim 4, characterized in that, The data block has a corresponding key value range, which overlaps with the value of the target key.

7. The method according to claim 4, characterized in that, The search range for the data block is defined between (V – E) and (V + E).

8. The method according to claim 1, characterized in that, The multiple data blocks, the PLA function, and the E are stored in persistent storage.

9. The method according to claim 4, characterized in that, Also includes: Before initiating the discovery operation to discover data blocks from the plurality of data blocks, a simplified index model is retrieved from persistent storage.

10. The method according to claim 4, characterized in that, Also includes: After the discovery operation for discovering the data block from the plurality of data blocks, the data block is retrieved from persistent storage.

11. The method according to claim 4, characterized in that, The discovery operation used to find the target key is a binary search or a linear search.

12. The method according to claim 4, characterized in that, Also includes: Before initiating the discovery operation to discover the target key, an indication of the most significant bit is requested from and received from the RL agent.

13. The method according to claim 12, characterized in that, The discovery operation for discovering the target key includes: selecting an instant key within the search range, and comparing each bit of a portion of the value of the instant key with the corresponding bit of the value of the target key. The portion of the value of the instant key is defined by the first bit and the most significant bit.

14. The method according to claim 4, characterized in that, Also includes: The RL agent records the delay of the lookup query request and adjusts the parameters for future lookup query requests.

15. A storage system, characterized in that, The storage system is a log-structured merge tree-based storage system, and the storage system includes electronic devices, wherein the electronic devices are used for: Provide multiple keys; A sorted string table (SST) is generated by arranging the key values ​​of the multiple keys in ascending order. The corresponding keys of the SST are indexed using a sequence of positive integers; Receive request; Request and receive the maximum block size value (Bmax) and model error value (E) from the reinforcement learning (RL) agent. The SST is divided into multiple data blocks, wherein each data block includes one or more corresponding keys; Generate a piecewise linear approximation (PLA) function; Store the plurality of data blocks, the PLA function, and the E.

16. The storage system according to claim 15, characterized in that, The electronic device is also used for: Receive an indication of a target key, wherein the indication is at least in part based on a pointer, implemented data, or a combination thereof; Receive a search query request to find the target key among the plurality of keys; Determine whether the value of the target key is within the range of key values ​​in the SST; If the value of the target key is within the range of the key values ​​of the SST, then a discovery operation is initiated to discover the data block from the plurality of data blocks; The value of the target key is input into the function, and the function is run to generate the value (V) of the center of the search range. The search range for generating the data block; Initiate a discovery operation to find the target key within the search range; Determine whether the target key has been found within the search range; If the target key is found within the search range, the target key is reported along with the corresponding pointer to the value block.

17. The storage system according to claim 16, characterized in that, The function is either the PLA function or the piecewise regression approximation (PRA) function.

18. The storage system according to claim 15, characterized in that, The sum of the values ​​of the one or more corresponding keys of each data block is less than or equal to Bmax.

19. The storage system according to claim 15, characterized in that, The first key of the first data block is the first instance of the first data block. The final key of each data block is the spline point of each data block. The PLA function is constructed based on the spline points of each data block and the first spline points of the first data block. The maximum interpolation error of the PLA function is less than or equal to E.

20. The storage system according to claim 16, characterized in that, The search range for the data block is defined between (V – E) and (V + E).

Images