Key-value database read-write optimization method and system based on CSD learning index

Abstract

This application relates to the field of key-value database technology, and discloses a method and system for optimizing key-value database read and write operations based on a CSD (Computer-Simplified Data Structure) learning index. The method includes: collecting key-value pairs to be persisted in memory; sorting them by key to form an ordered key sequence; and evaluating the access frequency of key intervals corresponding to the key sequence based on real-time access statistics. For key intervals with access frequency exceeding a preset threshold, training and utilizing the corresponding learning index model to output the predicted offset position of each key-value pair in the key sequence. Based on the predicted offset position, the key-value pairs are written to a storage device to form an ordered string table file, and the physical storage gaps in the predictive active layout are automatically compressed using a computational storage device. The method for constructing the newly generated ordered string table file is dynamically determined based on changes in access frequency, real-time system load, and the merging process of the LSM tree. This application significantly improves the efficiency of hot data reading and maintains a balance between read performance and system resource utilization.

Description

Technical Field

[0001] This application relates to the field of key-value database technology, and for example to a method and system for optimizing key-value database read and write operations based on CSD learning indexes. Background Technology

[0002] Key-value stores, as high-performance storage systems that organize data in a concise key-value pair format, have been widely used in numerous application scenarios, including cloud storage, big data analytics, content delivery networks, and blockchain storage, thanks to their flexible data models and excellent read / write throughput. To cope with high concurrency and high write loads, the industry widely adopts storage engines based on the LSM (Log-Structured Merge-tree) architecture. This significantly improves system write performance by converting random writes into in-memory batch caching and sequential disk writes, becoming the foundation for many current key-value storage systems.

[0003] In an LSM-based key-value store system, data is first written to a memory table (MemTable). Once full, it is converted into ordered string table files (SSTable) and managed in a hierarchical stack. To improve read efficiency, the system uses a Bloom filter to quickly determine whether a key might exist in a particular ordered string table file, and within the ordered string table file, it relies on a sparse index to locate data blocks. However, as data accumulates, the number and levels of ordered string table files increase, leading to a significant read amplification problem: a single query requires accessing multiple files at multiple levels, resulting in substantial I / O operations and search overhead. While Bloom filters can reduce invalid accesses, they cannot prevent cross-level lookups; sparse indexes, on the other hand, require repeated location of data blocks. To improve index efficiency, Learned Index (LI) technology is introduced, which replaces the traditional index structure with a machine learning model, mapping the key space to a location space to achieve efficient predictive retrieval. However, due to the compact and variable-length storage format of key-value pairs in traditional ordered string table files, the tight dependencies between data prevent random loading or efficient parallel prediction, limiting the practical application of learning indexes in key-value databases. Therefore, there is an urgent need for a key-value database read optimization method to solve the above problems.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments, but rather as a prelude to the detailed description that follows.

[0006] This disclosure provides a method and system for optimizing read and write operations of key-value databases based on CSD learning indexes, in order to solve the technical problems of limited read performance and poor compatibility with learning indexes in existing LSM tree key-value databases.

[0007] In some embodiments, a key-value database read / write optimization method based on CSD learned indexes is applied to a key-value database with an LSM tree architecture, and the method includes:

[0008] Collect key-value pairs to be persisted in memory, sort them by key to form an ordered key sequence, and evaluate the access frequency of the key interval corresponding to the key sequence based on real-time access statistics.

[0009] For key intervals with access popularity exceeding a preset threshold, a learning-based indexing model is trained based on the key sequence and the corresponding real offset position.

[0010] Using the trained learning index model, the predicted offset position of each key-value pair in the key sequence is output according to the key sequence. The learning index model is used to establish the mapping relationship between key value and offset position, and outputs the bounded predicted offset position with a one-way right offset.

[0011] Based on the predicted offset position, key-value pairs are written to the storage device to form an ordered string table file, thus completing the predictive active layout. The transparent compression mechanism of the compute storage device is used to automatically compress the physical storage gaps in the predictive active layout.

[0012] Based on changes in access popularity, real-time system load, and the merging process of the LSM tree, the system dynamically decides how to construct the newly generated ordered string table file. The construction method includes construction based on a trained learning index model or construction based on a traditional sequential layout.

[0013] In some embodiments, the learned indexing model is a piecewise order-preserving regression model, which is trained using the following optimization function:

[0014]

[0015] In the formula, For the learning index model of key The predicted offset position, For key The actual offset position, This represents the total number of key-value pairs in the key sequence. The parameters for the piecewise order-preserving regression model, The first term is the order-preserving penalty coefficient; the second term is the prediction error loss term; and the third term is the order-preserving constraint term.

[0016] In some embodiments, after outputting the predicted offset positions of each key-value pair in the key sequence, the method further includes:

[0017] Based on an ordered sequence of keys, key-value pairs are scanned sequentially, and the data block currently being built is monitored.

[0018] If the size of the current data block reaches the preset maximum block size, or if the error between the cumulative predicted offset position and the actual offset position is detected to exceed the preset maximum allowable error during the scanning process, the construction of the current data block will be terminated and a new data block will be created.

[0019] The process of repeatedly scanning, monitoring, and creating new data blocks when conditions are met continues until the entire key sequence is divided into one or more consecutive key-value data blocks, forming a data block set.

[0020] In some embodiments, the data block set is organized using a radix tree structure, including:

[0021] Each data block is mapped to a node in a radix tree. The node stores the key prefix, interval boundaries, and corresponding learned index model parameters of the corresponding data block.

[0022] During reading, the data block node containing the target key is located by traversing the cardinality tree, and the predicted offset position of the key within the data block is output using the learned index model stored in the corresponding node.

[0023] In some embodiments, the access popularity of a key interval corresponding to a key sequence is evaluated based on real-time access statistics, including:

[0024] The system counts read access events of the key interval from the memory layer and the disk layer, maintains a forward access counter and a reverse access counter for the key interval, and records the cumulative number of forward and reverse accesses respectively.

[0025] Based on the exponential decay method, the positive and negative heat values ​​at the current moment are calculated separately using the following formula:

[0026]

[0027]

[0028] In the formula, This is a positive heat value. This is the reverse heat value. , These represent the cumulative number of visits in the forward and reverse directions, respectively. , These are the timestamps of the most recent forward and reverse accesses, respectively. As the attenuation factor, This is the current timestamp;

[0029] The positive and negative heat values ​​are weighted and fused, and then normalized by dividing by the survival time of the key interval to obtain the comprehensive heat value. The calculation formula is as follows:

[0030]

[0031] In the formula, The overall popularity score, , These are the weighting coefficients. This is the starting time for statistics used to assess visit popularity.

[0032] In some embodiments, the construction method of the newly generated ordered string table file by dynamically deciding includes:

[0033] If the overall popularity value of the key interval is higher than the preset hot data threshold, then a training-based learning index model will be used to construct the index.

[0034] If the overall heat value of the key interval is lower than the preset cold data threshold, then the construction method based on the traditional sequential layout will be adopted.

[0035] If the overall heat value of the key interval represents medium-temperature data, then the decision on whether to adopt the training-based learning index model construction method is made based on the real-time load of the system.

[0036] In some embodiments, during the merging of LSM trees, the method further includes:

[0037] If the overall heat value of the key interval is higher than the preset hot data threshold, then the newly generated ordered string table file after merging will be constructed using a training-based learning index model.

[0038] If the overall heat value of the key interval is lower than the preset cold data threshold, the newly generated ordered string table file will be constructed using a traditional sequential layout method.

[0039] In some embodiments, writing key-value pairs to a storage device based on predicted offset positions to form an ordered string table file includes:

[0040] Write the predicted offset position as the corresponding key value to the physical storage start address within its data block.

[0041] In some embodiments, the physical storage gaps in a predictive active layout are automatically compressed using a transparent compression mechanism of the computing storage device, including:

[0042] In predictive active layout, the transparent compression function of the compute storage device is invoked to perform real-time compression processing on storage areas that are logically continuous but physically discontinuous, resulting from non-contiguous key-value pair layout. The compute storage device performs data compression and storage at the hardware layer and automatically decompresses the data during reading to maintain logical address continuity and eliminate physical storage gaps.

[0043] In some embodiments, a key-value database read / write optimization system based on CSD learned indexes is applied to a key-value database with an LSM tree architecture. The system includes:

[0044] The read popularity data statistics module is used to collect key-value pairs to be persisted in memory, sort them by key to form an ordered key sequence, and evaluate the access popularity of the key interval corresponding to the key sequence based on real-time access statistics.

[0045] The learning index building module is used to train a learning index model based on the key sequence and the corresponding real offset position for key intervals with access popularity higher than a preset threshold.

[0046] The predictive active layout module is used to utilize a trained learning index model to output the predicted offset position of each key-value pair in the key sequence based on the key sequence. The learned index model is used to establish the mapping relationship between key values ​​and offset positions, and outputs the bounded predicted offset position with a one-way right offset.

[0047] The predictive active layout module is also used to write key-value pairs based on the predicted offset position to complete the predictive active layout;

[0048] Storage components, including compute storage devices, are used to automatically compress physical storage gaps in the layout using the transparent compression mechanism of the compute storage devices;

[0049] The index building trade-off decision module is used to dynamically decide how to build the newly generated ordered string table file based on changes in access popularity, real-time system load, and the merging process of the LSM tree. The building method includes building based on a trained learning index model or building based on a traditional sequential layout.

[0050] The key-value database read / write optimization method and system based on CSD learning index provided in this disclosure can achieve the following technical effects:

[0051] This application first collects key-value pairs to be persisted and sorts them by key to form an ordered key sequence. Simultaneously, it assesses the access frequency of corresponding key intervals based on real-time access statistics. For key intervals with access frequency exceeding a preset threshold, a learning index model is trained based on the key sequence and its corresponding actual offset position. This model can output a bounded predicted offset position with a unidirectional rightward offset, effectively shortening the hot data location path. The reading process of this application does not require scanning multiple layers of ordered string tables layer by layer, nor does it rely on additional auxiliary structures. It only needs to scan a limited range to the right from the predicted offset output by the model to hit the target key, significantly improving the efficiency of hot data reading. Based on this, this application writes key-value pairs to a storage device based on the predicted offset position to form an ordered string table file, completing predictive active layout. Simultaneously, it utilizes the transparent compression mechanism of the computing storage device to automatically handle the physical storage gaps generated by the layout, effectively solving the learning index adaptation problem caused by the compact variable-length storage format of traditional ordered string tables. It achieves compatibility with the learning index without modifying the key-value encoding format of the ordered string table. The transparent compression of the computing storage device also eliminates the additional physical storage overhead caused by sparse layout, balancing adaptability and space utilization. Finally, this application combines changes in access popularity, real-time system load, and the LSM tree merging process to dynamically decide whether to build the newly generated ordered string table file based on a trained learning index model or based on a traditional sequential layout. This avoids the redundant computation and storage overhead caused by building a full learning index and allows the index layout to adapt to read and write loads during the merging process, further maintaining a balance between read performance stability and system resource utilization. This ensures that the entire key-value database maintains efficient read operation performance under different load scenarios.

[0052] The above general description and the description below are exemplary and illustrative only and are not intended to limit this application. Attached Figure Description

[0053] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations and drawings do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are shown as similar elements. The drawings are not to be scaled. And wherein:

[0054] Figure 1 This is a schematic diagram of a learning index architecture based on segmented order-preserving regression provided in an embodiment of this disclosure;

[0055] Figure 2 This is a flowchart illustrating a key-value database read / write optimization method based on a CSD learning index provided in an embodiment of this disclosure;

[0056] Figure 3 This is a schematic diagram of a learning index model construction strategy for hot and cold perception provided in an embodiment of this disclosure;

[0057] Figure 4 This is a schematic diagram of a key sequence data block partitioning and organization strategy based on learning index prediction provided in an embodiment of this disclosure;

[0058] Figure 5 This is a schematic diagram of the hot and cold range adaptation process of a learned index during an LSM tree merging process provided in an embodiment of this disclosure. Detailed Implementation

[0059] To provide a more detailed understanding of the features and technical content of the embodiments of this disclosure, the implementation of the embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for illustrative purposes only and are not intended to limit the embodiments of this disclosure. In the following technical description, for ease of explanation, several details are used to provide a full understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be simplified in their depiction to simplify the drawings.

[0060] The terms "first," "second," etc., used in the embodiments of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this disclosure described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion.

[0061] Unless otherwise stated, the term "multiple" means two or more.

[0062] In this embodiment of the disclosure, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0063] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0064] The term "correspondence" can refer to an association or binding relationship. The correspondence between A and B means that there is an association or binding relationship between A and B.

[0065] To reduce read latency, LSM trees rely on Bloom filters to quickly determine whether a key might exist in an ordered string table file, and then use a sparse index within the file to perform a binary search to locate the data block. However, as the system continues to run, the number and levels of ordered string table files accumulate, causing significant read amplification. While Bloom filters can reduce invalid accesses, they cannot prevent cross-level searches; sparse indexes require repeated location between data blocks. Therefore, in typical read-first or hotspot read scenarios, the read performance of the LSM tree architecture is often limited by the overhead of multi-level file searches and data location. To address the shortcomings of traditional index structures in terms of read performance, learning index technology, as an emerging data indexing method, has been proposed to optimize retrieval operations. Learning indexes replace traditional index structures with machine learning models, mapping the key space to a location space, and using data distribution patterns to predict locations, thereby reducing the maintenance cost of the index structure and improving query efficiency. However, applying learned indexes to ordered string table files in LSM trees presents challenges: due to the compact and variable-length storage format of key-value pairs in traditional ordered string table files, the tight dependencies between data make it impossible to achieve random loading or efficient parallel prediction, which limits the practical application of learned indexes in key-value databases.

[0066] To address the aforementioned issues, this disclosure provides a method and system for optimizing key-value database read / write operations based on CSD (Computational Storage Drive) learning indexes. With the development of new storage hardware, CSDs integrate internal computing units with high-performance storage media, supporting transparent compression and intelligent data processing. CSDs can automatically compress underlying data without altering the external access interface, significantly improving logical storage space utilization while allowing for higher sparsity tolerance in the underlying data layout. This provides new design space for data distribution-based structural optimization, enabling the application of learning indexes in key-value database storage systems at a lower cost.

[0067] Figure 1 This is a schematic diagram of a learning index architecture based on segmented order-preserving regression provided in an embodiment of this disclosure. Combined with... Figure 1The read / write optimization method of this application is applied to a key-value database with an LSM tree architecture. The core architecture is as follows: The memory layer contains a readable and writable memory table (MemTable) and a read-only immutable memory table (ImmMemTable). After the key-value pairs to be persisted are merged, they are organized into an ordered key sequence. A learned index model is trained based on this key sequence, and then the data blocks are reorganized using the model. The storage space adopts a hierarchical (L0 to Ln) storage architecture of an LSM tree. During the merging process, an ordered string table file is generated, which contains data blocks and a radix tree written in a lightweight serialized form. When reading data, the learned index table is traversed first to locate the target associated data block, and then the corresponding learned index model is called to output the predicted offset position of the key, thereby achieving efficient location of the target data.

[0068] In some embodiments, when building a learned indexing model, the key-value pairs (KV pairs) to be persisted need to be organized in memory first. The system sorts the KV pairs by ascending key order to form an ordered key sequence {key1, key2, ..., key...}. n}, and determine the position of each key in the logical storage structure, thereby obtaining the actual offset position of the key, and generating the corresponding target offset sequence {offset1, offset2, …, offset}. n These keys and their offsets constitute the basic data for model training.

[0069] This application's learned index building module employs a piecewise isotonic regression (PIR) method to train the learned index model. Specifically, the learned index model is a piecewise isotonic regression model used to establish the mapping relationship between key values ​​and offset positions, outputting a bounded predicted offset position with a unidirectional rightward shift. This model is trained using the following optimization function:

[0070]

[0071] In the formula, For the learning index model of key The predicted offset position, For key The actual offset position, This represents the total number of key-value pairs in the key sequence. The parameters for the piecewise order-preserving regression model, This is the order-preserving penalty coefficient.

[0072] The first term is the prediction error loss term, used to minimize the difference between the predicted position and the true offset, so that the model's output is as close as possible to the true offset position. The second term is the order-preserving constraint term, used to penalize any prediction that violates the monotonically increasing order; that is, when the predicted value of the next key is not greater than the predicted value of the previous key, this term will incur a loss. Since the key sequence to be trained is already arranged in ascending order, and the corresponding true offset sequence also shows a natural increasing trend, when the predicted position of the next key is less than the predicted position of the previous key, the order-preserving constraint term will output a value greater than 0, causing the total loss of the optimization function to increase. The model will automatically adjust the parameters during iterative training to reduce the loss, ultimately forcing the entire predicted offset sequence to maintain its non-decreasing characteristic. At the same time, this optimization function, through the directional constraint on the prediction error, further ensures that the predicted position of each key will not be less than its corresponding true position, avoiding the situation where the predicted offset shifts to the left. This ensures that there is no overlap between any KV pairs (the prediction falls into the region of the previous key), thus completely eliminating the prediction overlap problem between any two KV pairs, providing a theoretical guarantee for the subsequent search process that only scans to the right.

[0073] The following description, in conjunction with the accompanying drawings, illustrates the key-value database read / write optimization method based on CSD learning indexes provided in this disclosure.

[0074] Figure 2 This is a flowchart illustrating a key-value database read / write optimization method based on a CSD learning index provided in an embodiment of this disclosure. Figure 2 As shown, the method includes the following steps:

[0075] S201: Collect key-value pairs to be persisted in memory, sort them by key to form an ordered key sequence, and evaluate the access frequency of the key interval corresponding to the key sequence based on real-time access statistics.

[0076] In some embodiments, the system first collects key-value pairs (KV pairs) that meet preset persistence conditions (such as the memory table space reaching a threshold) from the MemTable and Immutable MemTable in the memory layer. These KV pairs will serve as the core foundational data for the subsequent construction of the ordered string table. To adapt to the learning requirements of the learning index model for key space and position mapping, the system sorts and organizes the collected KV pairs according to the natural ascending order of keys, forming an ordered key sequence {key1, key2, …, key}. n This ensures the order and continuity of the key space, providing a prerequisite for subsequent model calls and data layout.

[0077] In some embodiments, the access popularity of the key interval corresponding to the key sequence is dynamically and accurately evaluated based on real-time access statistics. Figure 3This is a schematic diagram of a hot / cold sensing learning index model construction strategy provided in an embodiment of this disclosure, combined with... Figure 3 Specifically, firstly, all read access events for the given key range are counted across the memory layer (memory table and immutable memory table) and the disk layer (ordered character count tables at each level of the LSM tree). A forward access counter and a reverse access counter are maintained in memory for this key range, recording the cumulative number of forward and reverse accesses, i.e., the cumulative number of accesses in ascending and descending order. Then, based on exponential decay and time decay characteristics, the current forward and reverse heat values ​​are calculated using the following formula:

[0078]

[0079]

[0080] In the formula, This is a positive heat value. This is the reverse heat value. , These represent the cumulative number of visits in the forward and reverse directions, respectively. , These are the timestamps of the most recent forward and reverse accesses, respectively. As the attenuation factor, This is the current timestamp. Finally, to uniformly measure the overall access popularity of this key interval, the positive and negative popularity values ​​are weighted and merged, then normalized by dividing by the key interval's lifespan (the difference between the current timestamp and the starting timestamp of the popularity statistics), resulting in a comprehensive popularity value. The corresponding calculation formula is as follows:

[0081]

[0082] In the formula, The overall popularity score, , This is a weighting coefficient used to adjust the importance of forward and reverse access in popularity assessment based on different business characteristics (such as scenarios with more writes than reads or primarily sequential reads). The statistical start time used to assess access popularity is used, while the lifespan of the key interval (denominator) can effectively reflect the average popularity per unit time during the life cycle of the key interval, avoiding abnormally high popularity caused by a large number of concentrated accesses in a short period of time. The final comprehensive popularity value can stably and objectively characterize the overall activity level of the key interval during its life cycle, providing an accurate decision basis for whether to build and update the learning index model.

[0083] S202: For key intervals with access popularity exceeding a preset threshold, a learning index model is trained based on the key sequence and the corresponding true offset position.

[0084] S203: Using the trained learning index model, output the predicted offset position of each key-value pair in the key sequence according to the key sequence, wherein the learned index model is used to establish the mapping relationship between key values ​​and offset positions, and outputs the bounded predicted offset position with a one-way right offset.

[0085] In some embodiments, for key intervals whose access popularity is determined to be higher than a preset threshold based on the comprehensive popularity value, the system will invoke a piecewise order-preserving regression (PIR) learned indexing model trained specifically for that key interval. This model has been trained with parameters through an optimization function and can stably establish an accurate mapping relationship between key values ​​and offset positions. During the invocation process, an ordered key sequence is used as the model input, and the predicted offset position corresponding to each key-value pair is calculated and output one by one. The predicted offset position output by the model is a one-way right offset, that is, through the synergistic effect of the order-preserving constraint term and the prediction error loss term in the optimization function, it is ensured that the predicted position of each key is always no less than its true position, and there will be no leftward offset. In addition, the predicted offset position is bounded, the deviation between the predicted offset and the true offset is controlled within a preset range, and the predicted offsets of adjacent keys maintain a non-decreasing characteristic to avoid prediction overlap between key-value pairs.

[0086] In some embodiments, after the learned indexing model outputs the predicted offset positions of each key-value pair in the key sequence, this application executes a data block partitioning process to ensure the orderliness of the subsequent storage layout and query efficiency. Specifically, this application uses the sorted key sequence as a basis, sequentially traversing each key-value pair while simultaneously monitoring two metrics of the currently constructed data block in real time: the current size of the data block and the cumulative error between the predicted and actual offset positions. Here, this application presets two thresholds: the maximum block size... With maximum allowable prediction error During the traversal, if the size of the current data block reaches or exceeds the preset maximum block size... Or the cumulative prediction error detected exceeds the preset maximum permissible error. The system will immediately terminate the construction of the current data block and create a new data block to accommodate subsequent key-value pairs. Subsequently, the process of sequential scanning, indicator monitoring, and conditional judgment is repeated until the entire ordered key sequence is completely divided into one or more consecutive key-value data blocks, ultimately forming a structured data block set R, R={Block1, Block2, …, Block…}. m The above measures, which avoid the decrease in search efficiency caused by excessively large single data blocks and ensure that the prediction accuracy within each data block remains within a controllable range, lay the foundation for subsequent efficient search through radix tree index organization.

[0087] In some embodiments, to achieve fast data block location and efficient index retrieval, this application employs a radix tree structure to organize and manage the aforementioned data block set. Specifically, each partitioned data block is mapped to an independent node in the radix tree. This node not only associates with the storage address information of the corresponding data block but also stores the key prefix within the corresponding data block, the interval boundaries corresponding to the data block (i.e., the minimum and maximum keys within the interval), and the model parameters of the piecewise order-preserving regression (PIR) learned index model trained for that data block. This enables the radix tree to achieve data block location and index model retrieval without the need for an additional independent index structure, effectively reducing storage overhead.

[0088] During subsequent data reading, the cardinality tree can be traversed based on the prefix information of the target key. Prefix matching can quickly locate the node corresponding to the data block containing the target key. After location, the learned index model stored in that node is invoked. By inputting the target key, its predicted offset position within the corresponding data block can be quickly output, allowing direct jump to the target storage area to complete data reading.

[0089] In addition, the radix tree structure is serialized in a lightweight structure and written to a new ordered string table file along with the data blocks, replacing the index blocks in the traditional ordered string table file, thus achieving compatibility with existing LSM storage frameworks.

[0090] The following section, in conjunction with the accompanying diagram, explains the strategy for partitioning and organizing key sequence data blocks based on learned index prediction.

[0091] Figure 4 This is a schematic diagram illustrating a key sequence data block partitioning and organization strategy based on learned index prediction, provided in an embodiment of this disclosure. Combined with... Figure 4 The process begins by inputting a sequence of keys to be persisted, sorted in ascending order. This sequence is then scanned sequentially while a corresponding learned indexing model is trained. During training, the model's prediction error is constrained to a unidirectional rightward offset. After obtaining the mapping between key values ​​and offset positions using the trained model, the key sequence is scanned again for partitioning. During the scan, the size of the current data block and the cumulative prediction error are checked to ensure they do not exceed a preset limit. If they do not exceed the limit, the scan continues. When any metric reaches the preset limit, the current block construction is terminated, and a new block is created, until the entire key sequence is partitioned, and the database collection is output. Finally, all partitioned data blocks are organized in a radix tree structure, with each data block mapped to a node in the tree, storing the key prefix, interval boundaries, and model parameters to support efficient subsequent lookups.

[0092] In some embodiments, the construction process of a learned index uses key-value pairs. Maximum permissible prediction error and maximum block size As input, output a set of Radix Tree nodes. Specifically, the system traverses the set of key-value pairs. For each key-value pair (k, v) in the data, the size of the current data block is continuously checked during the traversal. Is it less than and current cumulative prediction error Is it less than If both conditions are met, then the piecewise order-preserving regression model construction function is called. To train a learned indexing model The model optimizes the function:

[0093] Training begins. After model training is complete, the current key-value pair is added to the current data block, and the block size and error status are updated. If the size of the current data block or the cumulative prediction error exceeds a preset threshold, the data block is encapsulated into a radix tree node. This node stores the key prefix, interval boundaries, and learned index model parameters of the corresponding data block. Subsequently, the system creates new data blocks to continue processing the remaining key-value pairs. This process is repeated until all key-value pairs have been processed, ultimately generating a set of radix tree nodes. .

[0094] In other words, the set of KV pairs arranged in ascending order Maximum permissible prediction error Maximum block size As input, data blocks that meet threshold requirements are constructed by traversing key-value pairs: when the block size and error conditions are met, the key-value pairs are added to the current block and a piecewise order-preserving regression (PIR) learned index model is trained; otherwise, the current block is encapsulated as a radix tree node containing key prefixes, interval boundaries, and model parameters. Finally, a set of radix tree nodes is output, which not only achieves efficient index construction of hot data, but also is compatible with the original LSM storage framework, providing support for rapid location of target keys in the future.

[0095] S204: Write key-value pairs to the storage device based on the predicted offset position to form an ordered string table file, complete the predictive active layout, and use the transparent compression mechanism of the computing storage device to automatically compress the physical storage gaps in the predictive active layout.

[0096] In some embodiments, after calculating the predicted offset position of the key-value pair and dividing the data block, an ordered string table writing process is required. Since the model training stage has ensured that the predicted offset is always no less than the actual position through the optimization function, and the prediction error and block size are strictly controlled during the data block division process, the predicted offset position of each key-value pair is completely matched with the actual physical location to be stored, achieving the effect of "prediction is location". During writing, this application does not need to perform additional order adjustment or address correction on the key-value pairs. It directly uses the predicted offset position of each key-value pair as its physical storage starting address in the data block and writes it to the physical storage area corresponding to the data block in the computing storage device. The key-value pairs are linearly organized within the data block. The above writing method not only saves the storage overhead of the additional index structure in the data block of the traditional ordered string table, but also enables subsequent reading to directly jump to the target storage address through the model's predicted offset. This significantly reduces the complexity of intra-block lookup from the traditional O(log B) to close to O(1), significantly improving data positioning efficiency, while maintaining the compatibility of the ordered string table file format without modifying the underlying storage architecture.

[0097] In some embodiments, during the predictive active layout of key-value pairs based on predicted offset positions, the predicted offset positions of each key-value pair may be discontinuous, resulting in storage gaps that are logically continuous but physically discontinuous between the key-value pairs. This application addresses these storage gaps automatically by utilizing a transparent compression mechanism in the compute storage device. Specifically, the compute storage device monitors the distribution of storage areas in real time. When it detects physical gaps caused by discontinuous layout, it automatically compresses the storage of that area, transforming the sparse physical storage space into compact compressed data blocks. During subsequent read operations, the compute storage device automatically decompresses the compressed data blocks according to the logical address of the access request, ensuring that the upper-layer system still perceives a continuous logical address space.

[0098] S205: Based on the changes in access popularity, real-time system load, and the merging process of the LSM tree, dynamically decide on the construction method of the newly generated ordered string table file. The construction method includes construction based on a trained learning index model or construction based on a traditional sequential layout.

[0099] In some embodiments, this application uses a comprehensive popularity value as the core criterion, comparing it with preset hot data thresholds and cold data thresholds to dynamically decide how to construct the newly generated ordered string table file. The construction method includes construction based on a trained learning index model or construction based on a traditional sequential layout. Specifically, if the comprehensive popularity value is higher than the hot data threshold, it indicates that the key range (composed of the minimum and maximum keys of the newly generated file) is frequently accessed hot data. In this case, a learned index model will be trained for the newly generated ordered string table file using the learned index construction module, and the data layout will be completed based on this model. If the comprehensive popularity value is lower than the cold data threshold, it indicates that the key range is accessed very infrequently. To avoid unnecessary model training, storage, and maintenance overhead, the newly generated ordered string table file will be constructed using a traditional sequential layout, with key-value pairs written to disk in sequence. If the overall heat value is between the hot and cold thresholds, which is characterized as medium-temperature data, then a flexible decision will be made based on the real-time system load (such as CPU utilization, memory usage, I / O activity, etc.): when system resources are sufficient and the load is low, a learning index model will be trained for the newly generated ordered string table file and built based on the model; when the system load is high, the construction of the learning index model will be delayed, or a traditional sequential layout will be used to build a new file to avoid the index construction consuming resources and affecting the stability of read and write services.

[0100] In some embodiments, during the LSM tree compaction process, the compaction optimization decision module can make targeted decisions on the construction method of the newly generated ordered string table file after the merge based on the key interval popularity of the files to be merged. Specifically, the system first selects the set of ordered string table files to be merged, determines the key interval corresponding to the file set (composed of the minimum and maximum keys of the file set), and obtains its comprehensive popularity value. If the comprehensive popularity value is still higher than the hot data threshold, it indicates that it is still an access hotspot, and the system adopts a training-based learning index model to construct the newly generated ordered string table file after the merge. If the comprehensive popularity value is lower than the cold data threshold, it indicates that it has lost its access hotspot, and to reduce the compaction overhead and the storage burden of the new ordered string table, the newly generated ordered string table file after the merge adopts a construction method based on the traditional sequential layout. If the comprehensive popularity value represents medium-temperature data, the system makes decisions based on the real-time load: when the load is low, a training-based learning index model is trained for the new file and the file is built based on the model; when the load is high, the traditional sequential layout is used to avoid occupying core resources and affecting the stability of read and write services. This indexing approach, which works closely with the merging process, allows the index layout of the newly generated ordered string table to adapt to changes in read / write load, ensuring performance for reading hot data while minimizing maintenance costs for cold data.

[0101] Figure 5This is a schematic diagram illustrating the hot and cold range adaptation process of a learned index during an LSM tree merging process provided in this embodiment of the disclosure. Combined with... Figure 5 From the moment the merge process is triggered, the system first selects the ordered string table files to be merged, reads the files, and determines their corresponding key ranges (composed of the minimum and maximum keys of the file set), while simultaneously extracting the access frequency information of these key ranges. Then, based on the comprehensive frequency value, the system makes a classification decision for these key ranges: if the key range is a hot data range, the system will directly train a dedicated learning index model for the newly merged ordered string table file, and then execute predictive active layout based on this new model to complete the construction of the new file; if it is determined to be a cold data range, there is no need to train a learning index, and the traditional sequential layout is used directly to complete the construction of the new file, minimizing storage and maintenance overhead; if it is determined to be a medium-temperature data range, further consideration is needed based on the real-time system load: when the system load is low and resources are sufficient, the process of training the new learning index model and predictive active layout is executed to ensure data access performance; when the system load is high, to avoid index construction consuming core resources and affecting the stability of read and write services, the system will switch to traditional sequential layout to build the new file. The final merged new ordered string table file not only ensures high-performance reading of hot data through learning indexes, but also reduces the maintenance overhead of cold data through traditional layout, thus achieving a dynamic balance between read / write performance and system overhead.

[0102] Based on the same inventive concept as the aforementioned key-value database read / write optimization method based on CSD learned indexes, this application also discloses a key-value database read / write optimization system based on CSD learned indexes in some embodiments, applied to LSM tree-structured key-value databases. This system includes: a read popularity data statistics module, used to collect key-value pairs to be persisted in memory, sort them by key to form an ordered key sequence, and evaluate the access popularity of the key intervals corresponding to the key sequence based on real-time access statistics; a learned index construction module, used to train a learned index model based on the key sequence and the corresponding real offset position for key intervals with access popularity higher than a preset threshold; and a predictive active layout module, used to utilize the trained learned index model to optimize key read / write operations based on key... The sequence output provides the predicted offset positions of each key-value pair in the key sequence. The learned indexing model establishes a mapping relationship between key values ​​and offset positions, outputting bounded predicted offset positions with a unidirectional rightward offset. The predictive active layout module is also used to write key-value pairs based on the predicted offset positions to complete the predictive active layout. A storage component, including a computing storage device, is used to automatically compress physical storage gaps in the layout using the transparent compression mechanism of the computing storage device. An index construction trade-off decision module dynamically decides the construction method of the newly generated ordered string table file based on changes in access popularity, real-time system load, and the merging process of the LSM tree. The construction method includes construction based on a trained learned indexing model or construction based on a traditional sequential layout.

[0103] The technical solutions of this disclosure can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes one or more instructions executed by a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in this disclosure. The aforementioned storage medium can be a non-transitory storage medium, including: USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, and other media capable of storing program code; it can also be a transient storage medium.

[0104] The foregoing description and accompanying drawings fully illustrate embodiments of this disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, procedural, and other changes. The embodiments represent only possible variations. Individual components and functions are optional unless explicitly required, and the order of operation may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terminology used in this application is for descriptive purposes only and is not intended to limit the scope of protection. As used in the description herein, the singular forms “a,” “an,” and “the” are intended to equally include the plural forms unless the context clearly indicates otherwise. Similarly, the term “and / or” as used herein means including one or more of the associated listed items and all possible combinations thereof. Additionally, when used in this application, the term "comprise" and its variations "comprises" and / or "comprising" refer to the presence of stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Without further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, or apparatus that includes said element. In this document, each embodiment may focus on the differences from other embodiments, and similar or identical parts between embodiments can be referred to mutually. For methods, products, etc., disclosed in the embodiments, if they correspond to the method section disclosed in the embodiments, the relevant parts can be referred to the description of the method section.

[0105] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this disclosure. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0106] The methods and products (including but not limited to devices and equipment) disclosed in the embodiments herein can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units may be merely a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to implement this embodiment according to actual needs. In addition, the functional units in the embodiments of this disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Claims

1. A key-value database read / write optimization method based on CSD learning index, applied to LSM tree-structured key-value databases, characterized in that, The method includes: Collect key-value pairs to be persisted in memory, sort them by key to form an ordered key sequence, and evaluate the access frequency of the key interval corresponding to the key sequence based on real-time access statistics. For key intervals with access popularity exceeding a preset threshold, a learning index model is trained based on the key sequence and the corresponding actual offset position. Using a trained learning index model, the predicted offset position of each key-value pair in the key sequence is output based on the key sequence. The learned index model is used to establish a mapping relationship between key values ​​and offset positions, and outputs a bounded predicted offset position that shifts unidirectionally to the right. Based on the predicted offset position, key-value pairs are written to the storage device to form an ordered string table file, completing the predictive active layout, and the physical storage gaps in the predictive active layout are automatically compressed using the transparent compression mechanism of the computing storage device. Based on the changes in access popularity, real-time system load, and the merging process of the LSM tree, the system dynamically decides on the construction method of the newly generated ordered string table file. The construction method includes construction based on a trained learning index model or construction based on a traditional sequential layout.

2. The key-value database read / write optimization method based on CSD learning index as described in claim 1, characterized in that, The learned indexing model is a piecewise order-preserving regression model, which is trained using the following optimization function: In the formula, For the learning index model of key The predicted offset position, For key The actual offset position, This represents the total number of key-value pairs in the key sequence. The parameters for the piecewise order-preserving regression model, The first term is the order-preserving penalty coefficient; the second term is the prediction error loss term; and the third term is the order-preserving constraint term.

3. The key-value database read / write optimization method based on CSD learning index as described in claim 1, characterized in that, After outputting the predicted offset positions of each key-value pair in the key sequence, the method further includes: Based on an ordered sequence of keys, key-value pairs are scanned sequentially, and the data block currently being built is monitored. If the size of the current data block reaches the preset maximum block size, or if the error between the cumulative predicted offset position and the actual offset position is detected to exceed the preset maximum allowable error during the scanning process, the construction of the current data block will be terminated and a new data block will be created. The process of repeatedly scanning, monitoring, and creating new data blocks when conditions are met continues until the entire key sequence is divided into one or more consecutive key-value data blocks, forming a data block set.

4. The key-value database read / write optimization method based on CSD learning index as described in claim 3, characterized in that, The data block set is organized using a radix tree structure, including: Each data block is mapped to a node in a radix tree, and the node stores the key prefix, interval boundaries, and corresponding learned index model parameters of the corresponding data block. During reading, the data block node containing the target key is located by traversing the cardinality tree, and the predicted offset position of the key within the data block is output using the learned index model stored in the corresponding node.

5. The key-value database read / write optimization method based on CSD learning index as described in claim 1, characterized in that, The assessment of the access popularity of the key interval corresponding to the key sequence based on real-time access statistics includes: The system counts read access events of the key interval from the memory layer and the disk layer, maintains a forward access counter and a reverse access counter for the key interval, and records the cumulative number of forward and reverse accesses respectively. Based on the exponential decay method, the positive and negative heat values ​​at the current moment are calculated separately using the following formula: In the formula, This is a positive heat value. This is the reverse heat value. , These represent the cumulative number of visits in the forward and reverse directions, respectively. , These are the timestamps of the most recent forward and reverse accesses, respectively. As the attenuation factor, This is the current timestamp; The positive and negative heat values ​​are weighted and fused, and then normalized by dividing by the survival time of the key interval to obtain the comprehensive heat value. The calculation formula is as follows: In the formula, The overall popularity score, , These are the weighting coefficients. This is the starting time for statistics used to assess visit popularity.

6. The key-value database read / write optimization method based on CSD learning index as described in claim 5, characterized in that, The methods for constructing newly generated ordered string table files through dynamic decision-making include: If the overall popularity value of the key interval is higher than the preset hot data threshold, then a training-based learning index model will be used to construct the index. If the overall heat value of the key interval is lower than the preset cold data threshold, then the construction method based on the traditional sequential layout will be adopted. If the overall heat value of the key interval represents medium-temperature data, then the decision on whether to adopt the training-based learning index model construction method is made based on the real-time load of the system.

7. The key-value database read / write optimization method based on CSD learning index as described in claim 5, characterized in that, The method further includes the following steps during the LSM tree merging process: If the overall heat value of the key interval is higher than the preset hot data threshold, then the newly generated ordered string table file after merging will be constructed using a training-based learning index model. If the overall heat value of the key interval is lower than the preset cold data threshold, the newly generated ordered string table file will be constructed using a traditional sequential layout method.

8. The key-value database read / write optimization method based on CSD learning index according to claim 1, characterized in that, The step of writing key-value pairs to a storage device based on the predicted offset position to form an ordered string table file includes: The predicted offset position is written as the physical storage start address within its corresponding key-value pair.

9. The key-value database read / write optimization method based on CSD learning index according to claim 1, characterized in that, The automatic compression of physical storage gaps in predictive active layout using the transparent compression mechanism of computing storage devices includes: In the predictive active layout, the transparent compression function of the computing storage device is invoked to perform real-time compression processing on the storage areas generated by the non-contiguous key-value pair layout, which have continuous logical addresses but discontiguous physical addresses. The computing storage device performs data compression and storage at the hardware layer and automatically decompresses it during reading to maintain logical address continuity and eliminate physical storage gaps.

10. A key-value database read / write optimization system based on CSD learning indexes, applied to LSM tree-structured key-value databases, characterized in that... The system includes: The read popularity data statistics module is used to collect key-value pairs to be persisted in memory, sort them by key to form an ordered key sequence, and evaluate the access popularity of the key interval corresponding to the key sequence based on real-time access statistics. The learning index building module is used to train a learning index model based on the key sequence and the corresponding real offset position for key intervals with access popularity higher than a preset threshold. The predictive active layout module is used to utilize a trained learning index model to output the predicted offset position of each key-value pair in the key sequence based on the key sequence. The learned index model is used to establish a mapping relationship between key values ​​and offset positions, and outputs a bounded predicted offset position that shifts unidirectionally to the right. The predictive active layout module is also used to write key-value pairs based on the predicted offset position to complete the predictive active layout; Storage components, including compute storage devices, are used to automatically compress physical storage gaps in the layout using the transparent compression mechanism of the compute storage devices; The index building trade-off decision module is used to dynamically decide how to build the newly generated ordered string table file based on the changes in access popularity, real-time system load, and the merging process of the LSM tree. The building method includes building based on a trained learning index model or building based on a traditional sequential layout.

Images