Learning-based compression method and system for LSM tree based on computing storage device
By performing LSM-tree compression and index training in parallel on a compute storage device (CSD), the problems of model failure and serial dependency are solved, achieving efficient data management and query performance, and optimizing the write and read performance of LSM-tree.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2026-04-17
- Publication Date
- 2026-07-21
AI Technical Summary
In existing LSM-tree compression operations, model failure and retraining overhead are significant. The serial dependency of the PLR algorithm limits the training speed on parallel hardware, resulting in system performance bottlenecks and redundant data access. It fails to effectively combine compression and index training.
A collaborative hierarchical storage architecture is adopted, which divides the LSM tree into a hot data layer and a cold data layer. The computing storage device (CSD) is used to perform compression and index training in parallel. The learning index is trained on the CSD through the parallel piecewise linear regression algorithm (PPLR). Combined with pipelined processing and collaborative search mechanism, data merging and model training are synchronized.
It enables parallel execution of model training and data merging, hides the retraining time overhead, reduces host CPU load, improves system throughput and read performance, and optimizes read performance under write-intensive loads.
Smart Images
Figure CN122431600A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer storage system technology, and more specifically, to an LSM tree learning compression method and system based on computing storage devices. Background Technology
[0002] With the rapid development of internet services, big data analytics, and real-time applications, key-value (KV) storage systems have become the infrastructure of modern data centers. To cope with the demand for large-scale data writing, storage engines based on log-structured merge-tree (LSM-tree) structures (such as LevelDB and RocksDB) are widely used.
[0003] LSM-tree is a data structure optimized for write-intensive workloads. It transforms random writes into sequential writes by buffering write operations in memory (MemTable) and batch-flushing data to an immutable ordered string table (SSTable) on disk, significantly improving write throughput. The SSTable is organized into multiple levels, with the level capacity growing exponentially.
[0004] However, this design requires a trade-off between read and write performance. Read operations may require searching memory tables and multiple levels of SSTables, leading to high read latency (read amplification). To mitigate read amplification and reclaim invalid space, LSM-trees must periodically perform background compaction operations, merging and sorting high-level SSTables with low-level data, cleaning up invalid key-value pairs, and rewriting. Compaction operations are computationally and I / O intensive, consuming significant CPU resources and storage bandwidth, often resulting in write stalls and write amplification issues.
[0005] Learning indexes utilize machine learning models (typically linear regression models) to fit the cumulative distribution function of keys, thereby predicting the physical location of keys in the stored file. Piecewise Linear Regression (PLR) is widely used due to its ability to guarantee error bounds. While applying learning indexes to LSM-trees can significantly reduce disk accesses, current techniques have significant drawbacks when combining the two: First, model failure and retraining are costly. The compression operation of LSM-trees alters the physical layout of data within the SSTable, rendering older index models ineffective. Existing solutions typically treat compression and model training as two separate processes, meaning data needs to be read twice (once for compression, once for training), or the processed data needs to be moved from storage back to the host CPU for training. This not only increases data movement overhead but also causes model updates to lag; in write-intensive scenarios, query requests often have to fall back to the slow traditional index path.
[0006] Secondly, the PLR algorithm suffers from serial dependency. Traditional PLR algorithms, when calculating the slope of a line segment and determining the segmentation point, strictly depend on the state of the previous data point (i.e., point dependency and slope dependency). This inherent serial characteristic makes the algorithm difficult to execute efficiently on parallel hardware such as multi-core CPUs or FPGAs, severely limiting the training speed of models on large-scale datasets.
[0007] Computational Storage Device (CSD) is a new type of storage device that integrates computing units (such as FPGAs and ARM cores), allowing computation to be performed directly at the data storage location. CSD utilizes the high bandwidth within the storage device to reduce data movement between the host and the device via the PCIe bus and alleviate the load on the host CPU.
[0008] Currently, CSDs are used to accelerate simple tasks such as database scanning, filtering, and compression. However, for complex data structures like LSM-trees, most existing CSD solutions only focus on offloading the compression task to the device, failing to effectively address the index building problem. Simply offloading compression and index training separately still cannot solve the serial bottleneck at the algorithm level, nor can it eliminate redundant data access between the two. Therefore, there is an urgent need for a hardware-software co-design that can fully utilize the characteristics of CSDs to deeply integrate the LSM-tree compression process with the training process of learned indexes, in order to achieve high-performance in-storage data management.
[0009] The relevant technical terms in this field are as follows: LSM-tree (Log-Structured Merge-tree): A hierarchical, ordered, disk-oriented data structure that optimizes write performance by converting random writes into sequential writes.
[0010] SSTable (Sorted String Table): A sorted string table is the data storage file format of LSM-tree on disk, containing ordered key-value pairs.
[0011] Compaction: A background maintenance operation of LSM-tree, used to merge SSTables at different levels, discard invalid data, and reorder them to reduce read amplification and reclaim space.
[0012] Learned Index: A learned index is a technique that uses machine learning models (such as regression models) to replace traditional structures (such as B-trees) for indexing data, predicting data location by learning the data distribution.
[0013] PLR (Piecewise Linear Regression): A piecewise linear regression algorithm that uses multiple line segments to approximate the distribution of data. It is often used as the underlying model for building learned indexes. Summary of the Invention
[0014] To address the shortcomings of existing technologies, the purpose of this invention is to provide an LSM tree learning-based compression method and system based on computing storage devices.
[0015] According to the present invention, an LSM tree-learning compression method based on computing storage devices includes: Step S1: Construct a collaborative hierarchical storage architecture, dividing the LSM tree into a hot data layer and a cold data layer, wherein the hot data layer resides on the host side and the cold data layer resides on the compute storage device (CSD). Step S2: When the compression operation of the cold data layer is triggered, the host constructs a compression task descriptor and offloads it to the computing storage device CSD; Step S3: The computing storage device CSD performs a learning compression operation, simultaneously training and generating a new learning index model while merging and sorting the stored SSTable data files. Step S4: The computing storage device (CSD) writes the merged data to the storage medium and sends the trained index model parameters back to the host for metadata update.
[0016] Preferably, step S3 includes: Step S3.1: Start the pipeline processing mechanism to read the key-value data of the level to be compressed from the storage medium in blocks and put it into the internal buffer of the computing storage device CSD; Step S3.2: During the pipeline execution phase, the key-value data is sorted and deduplicated using a comparator, and a new SSTable data is generated using a key-value merger. Step S3.3: While performing step S3.2, the merged ordered key sequence is streamed into the model training accelerator. The ordered nature of the key sequence is used to train the piecewise linear regression model in parallel, so that the model training delay is masked by the I / O delay of data merging.
[0017] Preferably, step S3.3 employs the parallel piecewise linear regression (PPLR) algorithm, specifically including: Step S3.3.1: Divide the input key sequence into several batches of fixed size based on the distribution characteristics of the keys; Step S3.3.2: Within each batch, the first data point is forcibly fixed as the support point of the batch to eliminate the serial dependency between points, and the slope range of all keys in the batch relative to the support point is calculated in parallel by hardware. Step S3.3.3: Update the slope range of the line segment sequentially, use the reprocessing queue to temporarily store and prioritize the processing of the remaining subsequent keys due to the termination of the current line segment in the next batch, so as to ensure the continuity of line segments across batches.
[0018] Preferably, the method further includes: Step S5: The host receives a query request. If the query is not found in the hot data layer, the host uses the learned index model cached on the host to predict the physical address range of the target key on the computing storage device (CSD).
[0019] Preferably, the method further includes: step S6: the host sends a batch of query requests containing SSTable identifiers and predicted address ranges to the computing storage device CSD.
[0020] Preferably, the method further includes: step S7: the computing storage device (CSD) uses an onboard search engine to directly perform scanning and filtering within the storage medium based on the received address range, and only returns the successfully matched key-value pairs to the host.
[0021] According to the present invention, an LSM tree learning compression system based on a computing storage device includes: Host-side module: Used to manage the hot data layer and global metadata of the LSM tree, and responsible for generating compression tasks and processing address prediction for query requests; Computational Storage Device (CSD) Module: Connected to the host module via a bus, it stores the cold data layer of the LSM tree and integrates a programmable logic unit to perform offloaded computational tasks; Learning-based compression engine: Deployed in the computing storage device CSD module, configured to perform SSTable merge operations and parallel training of the index model; Collaborative indexing framework: configured to maintain data distribution consistency between the host module and the computing storage device (CSD) module, ensuring data flow and model updates are synchronized between storage tiers.
[0022] Preferably, the learning-based compression engine includes: The data loading unit is used to manage the dual buffers inside the computing storage device (CSD) to achieve overlap between reading data from flash memory and pipeline processing; The execution pipeline unit contains comparators, key-value mergers, and model training accelerators that work in parallel, enabling simultaneous data compression and model building in a single data traversal. The data writing unit is used to persistently store the generated new SSTable data and upload the generated model parameters to the host module via a high-speed bus.
[0023] Preferably, the model training accelerator is configured to hardware-encode the parallel piecewise linear regression (PPLR) algorithm, including: Batch processing logic unit, used to divide the key stream into independent batches and assign fixed support points; The parallel slope calculation unit utilizes the parallelism of the field-programmable gate array (FPGA) to simultaneously calculate the upper and lower bound slopes of multiple key values within a batch. The state update and rollback unit is used to detect error limits and push the remaining data into the reprocessing queue when the line segment termination is detected.
[0024] Preferably, the computing storage device (CSD) module further includes: An onboard search engine is configured to parse batch query requests sent by the host module. The onboard search engine utilizes the high bandwidth of the computing storage device (CSD) to directly read data blocks from the storage medium, performs in-situ scanning based on the predicted range, and filters invalid data based on key-value matching results to reduce the amount of data transfer between the host and the device.
[0025] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a learning-based compression pipeline within CSD, enabling model training and data merging to be executed in parallel, with the training speed matching the merging speed. This completely hides the time overhead of model retraining and ensures the real-time performance of the indexing model.
[0026] 2. By offloading data-intensive compression and scanning tasks to the CSD, this invention significantly reduces invalid data movement on the PCIe bus and lowers the load on the host CPU.
[0027] 3. The PPLR algorithm and hardware accelerator proposed in this invention break the serial dependency limitation of the traditional PLR algorithm, make full use of the parallel computing capability of FPGA, and significantly improve the system throughput.
[0028] 4. This invention combines a collaborative hierarchical architecture and a collaborative search mechanism, which not only ensures low-latency access to hot data but also achieves efficient retrieval of cold data, maintaining excellent read performance even under write-intensive loads. Attached Figure Description
[0029] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall modular architecture of the system in this embodiment of the invention; Figure 2 This is a schematic diagram of the pipeline structure of the learning compression engine in an embodiment of the present invention; Figure 3 This is a schematic diagram of the logical flow of the Parallel Piecewise Linear Regression (PPLR) algorithm in an embodiment of the present invention; Figure 4 This is a schematic diagram of the data flow for collaborative search in an embodiment of the present invention. Detailed Implementation
[0030] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.
[0031] Example 1 This embodiment provides an LSM tree learning compression method based on computing storage devices, including the following steps.
[0032] Step S1: Construct a collaborative tiered storage architecture; like Figure 1 As shown, the LSM tree is divided into a hot data layer and a cold data layer. The hot data layer resides on the host side, and the cold data layer resides on the compute storage device (CSD).
[0033] The hot data layer includes the MemTable (memory table) and layers 0 and 1 in the hierarchical structure. This part of the data is relatively small in volume and frequently accessed, residing in the host's memory. The host maintains a global metadata tree and a learning index model corresponding to all layers.
[0034] The cold data layer includes a second layer and deeper layers, with the data volume growing exponentially, residing on the compute storage device (CSD). The CSD is equipped with a flash memory controller, a NAND flash memory array, and a computing unit (FPGA in this embodiment, but ARM cores or ASICs can also be used), and is connected to the host via a PCIe bus.
[0035] Step S2: Unload the compression task; When a compression operation is triggered in the cold data layer, the host constructs a compression task descriptor and offloads it to the compute storage device (CSD). This compression task descriptor contains information such as the identifiers of the SSTable files to be merged and is sent via a custom NVMe command.
[0036] Step S3: Perform learning-based compression; CSD performs a learning-based compression operation, simultaneously training and generating a new learning index model while merging and sorting the stored SSTable data files. This step further includes: Step S3.1: Initiate the pipelined processing mechanism. Read key-value data of the level to be compressed from the storage medium in blocks into the CSD's internal buffer. A dual-buffer mechanism is used; while the engine is processing data in one buffer, the DMA controller prefetches the next batch of SSTable data into the other buffer in parallel.
[0037] Step S3.2: Merge Sort and Deduplication. During the pipeline execution phase, a comparator is used to perform multi-way merge sort on the key-value data, outputting a globally ordered sequence of key-value pairs; the key-value merger is used to process deletion markers, discard old version data, and deduplicate, generating new SSTable data.
[0038] Step S3.3: Parallel Training of the Learned Index. While executing step S3.2, the merged ordered key sequence is streamed into the model training accelerator. The ordered nature of the key sequence is used to train the piecewise linear regression model in parallel, allowing the model training latency to be masked by the I / O latency of data merging. Specifically, the Parallel Piecewise Linear Regression (PPLR) algorithm is used, including: Step S3.3.1: Divide the input key sequence into several batches of fixed size according to the distribution characteristics of the keys.
[0039] Step S3.3.2: Within each batch, the first data point is forcibly fixed as the support point of the batch, eliminating the serial dependency between points, and the slope range of all keys in the batch relative to the support point is calculated in parallel by hardware.
[0040] Step S3.3.3: Update the slope range of the line segment sequentially, use the reprocessing queue to temporarily store and prioritize the processing of the remaining subsequent keys due to the termination of the current line segment in the next batch, so as to ensure the continuity of line segments across batches.
[0041] Step S4: Persistence and parameter return; CSD writes the merged data to the storage medium and sends the trained index model parameters (slope, intercept) back to the host for metadata updates. The host then updates the index structure in memory accordingly.
[0042] This embodiment also includes a collaborative search step to optimize the read path, specifically including the following steps: Step S5: The host receives the query request. If there is no match in the hot data layer, the host uses its cached learned index model to predict the physical address range of the target key on the CSD. The prediction is achieved by calculating a linear regression model (position = key × slope + intercept).
[0043] Step S6: The host sends a batch of query requests containing SSTable identifiers and predicted address ranges to the CSD. This request is encapsulated as a custom NVMe command, namely a collaborative search request.
[0044] Step S7: The CSD utilizes its onboard search engine to perform scanning and filtering directly within the storage medium based on the received address range, returning only the successfully matching key-value pairs to the host. The CSD's controller (ARM core) parses the request, and the onboard search engine (FPGA) directly reads data within the specified address range from the flash memory into an internal buffer, performs scanning and comparison, and returns only the matching values.
[0045] Example 2 like Figure 1 As shown, this embodiment provides an LSM tree learning compression system based on a computing storage device. The system consists of two hardware components: a host and a computing storage device (CSD). The host is equipped with a CPU and DRAM, and is connected to the CSD via a PCIe bus. The CSD is equipped with a flash memory controller, a NAND flash memory array, and an onboard computing unit (in this embodiment, an FPGA is used, but an ARM core or ASIC can also be used).
[0046] In terms of software logic, this embodiment adopts a collaborative hierarchical storage strategy, specifically including: Hot data management (Host side): The MemTable (memory table) of the LSM tree, as well as levels 0 and 1 in the hierarchical structure, are defined as hot data layers. Because this data is relatively small in volume and accessed extremely frequently, it resides in host memory. The host side maintains the global metadata tree and the corresponding learned index model for all levels.
[0047] Cold Data Management (CSD Side): Level 2 and deeper levels of the LSM tree are defined as cold data layers. As the layers deepen, the amount of data grows exponentially. This data is physically stored in the CSD, and its heavy workload of compression and indexing model retraining is offloaded to the CSD for execution.
[0048] like Figure 2 As shown, this invention implements a hardware-based learning compression engine on a CSD FPGA. This engine aims to solve the I / O bottleneck problem caused by the traditional approach of "reading data first for compression, then reading data for training." The engine adopts a streaming processing architecture and specifically includes the following modules and steps: Data Loading Module: This module is responsible for reading the SSTable data to be merged. To hide read latency, this module employs a dual-buffer mechanism. It sets up two large on-chip buffers (e.g., Buffer A and Buffer B). While the engine is processing data in Buffer A, the DMA controller prefetches the next batch of SSTable data into Buffer B in parallel.
[0049] Execution Pipeline Module: This is the core processing unit, containing three sub-modules that work in parallel: Multiplexer: Receives multiple key-value streams from the input buffer, performs a multiplexed merge sort algorithm, and outputs a globally ordered sequence of key-value pairs; Key-value merger: processes the semantic logic of the LSM tree, such as handling the deletion of tags, discarding old version data, and deduplicating key-value pairs, generating a new SSTable data stream; Model training accelerator: This is the key innovation of this invention. It directly listens to the ordered key stream output by the comparator via an internal bus (such as AXIStream). While the key-value merging unit generates the SSTable, the accelerator uses the same data stream to train the piecewise linear regression model in parallel.
[0050] Since model training is performed on compressed data streams, no additional I / O reads are required, and the pipeline characteristics of FPGAs make the training speed lower than the read / write latency of flash memory, the latency of model training is completely hidden.
[0051] Data writing module: This module is responsible for writing the merged new SSTable data into the new space. At the same time, after the model training of an SSTable is completed, a set of extremely lightweight linear model parameters (slope, intercept) generated by the training will be packaged and sent back to the host via the PCIe bus, and the host will update the index structure in memory accordingly.
[0052] Traditional PLR algorithms suffer from severe data dependency: when calculating the range of the slope of the line segment at point i, it is necessary to know the data at point i. The state of a single point (whether a new line segment is started, the slope range of the current line segment, etc.) prevents the algorithm from being parallelized on an FPGA by unrolling loops. To address this issue, this invention designs the PPLR algorithm and a corresponding FPGA hardware accelerator, such as... Figure 3 As shown, the specific processing flow is as follows: Data batching: The input ordered key stream is divided into fixed-size batches, for example, each batch contains 512 keys.
[0053] Fixed support point: Within each batch, the PPLR algorithm forces the first bond of that batch to be used as the support point for subsequent slope calculations.
[0054] Parallel slope calculation: Utilizing the parallel characteristics of the FPGA, the upper and lower bound slopes of the remaining 511 bonds in the batch relative to the support point are calculated in parallel. This step eliminates the serial calculation dependency between points.
[0055] State update and line segment termination: After obtaining the slope of all points, a fast pipeline logic sequence checks whether these slopes meet the preset error limit. Once it is found that the slope of a certain point causes the current line segment to be unable to cover the point, the current line segment is determined to terminate, and a new line segment is started from that point.
[0056] Reprocessing queue mechanism: This is crucial for ensuring algorithm correctness. When a line segment is not yet finished in the current batch (i.e., it crosses the batch boundary), or when a new line segment has just begun at the end of a batch, the relevant data for that line segment (such as the remaining keys or the current slope state) is placed in a reprocessing queue. When processing the next batch, the accelerator prioritizes reading data from this queue and merges it with the data from the new batch. This ensures the continuity of line segments across batches, making the model generated by the parallel algorithm consistent with that of the serial algorithm.
[0057] This invention optimizes not only the write path (compression) but also the read path (query). For example... Figure 4 As shown, the collaborative search process is as follows: Host-side prediction: When a client query request is received, the host first checks the MemTable and levels 0 and 1. If no match is found, the host queries the learned index model in memory. The model predicts the range of physical offsets of the key in the SSTable on the CSD side by calculating a linear regression model (position = key × slope + intercept).
[0058] Constructing a query request: Instead of directly initiating read I / O, the host constructs a custom NVMe command, namely a collaborative search request. This request includes: the target SSTable ID, the predicted physical address range, and the target key.
[0059] CSD-side search: The CSD's controller (ARM core) parses the request and uses the onboard search engine (FPGA) to directly read the specified data from the flash memory into an internal buffer. The search engine then quickly scans and compares within the buffer to find the target key.
[0060] Return results: If the target key is found, CSD returns the value corresponding to that key via the PCIe bus (usually only tens to thousands of bytes). If not found (possibly due to a Bloom filter misjudgment or prediction error), CSD returns a miss status, and the host then performs further processing (such as continuing to search the next layer).
[0061] Compared to the traditional method (reading all data back to the host memory and then searching by the CPU), this invention greatly reduces the amount of data transferred on the PCIe bus, avoids I / O bandwidth becoming a performance bottleneck, and also reduces the decompression and search burden on the host CPU.
[0062] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.
[0063] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.
Claims
1. A LSM tree-learning compression method based on computing storage devices, characterized in that, include: Step S1: Construct a collaborative hierarchical storage architecture, dividing the LSM tree into a hot data layer and a cold data layer, wherein the hot data layer resides on the host side and the cold data layer resides on the compute storage device (CSD). Step S2: When the compression operation of the cold data layer is triggered, the host constructs a compression task descriptor and offloads it to the computing storage device CSD; Step S3: The computing storage device CSD performs a learning compression operation, simultaneously training and generating a new learning index model while merging and sorting the stored SSTable data files. Step S4: The computing storage device (CSD) writes the merged data to the storage medium and sends the trained index model parameters back to the host for metadata update.
2. The LSM tree-learning compression method based on computing storage devices according to claim 1, characterized in that, Step S3 includes: Step S3.1: Start the pipeline processing mechanism to read the key-value data of the level to be compressed from the storage medium in blocks and put it into the internal buffer of the computing storage device CSD; Step S3.2: During the pipeline execution phase, the key-value data is sorted and deduplicated using a comparator, and a new SSTable data is generated using a key-value merger. Step S3.3: While performing step S3.2, the merged ordered key sequence is streamed into the model training accelerator. The ordered nature of the key sequence is used to train the piecewise linear regression model in parallel, so that the model training delay is masked by the I / O delay of data merging.
3. The LSM tree-learning compression method based on computing storage devices according to claim 2, characterized in that, Step S3.3 employs the parallel piecewise linear regression (PPLR) algorithm, specifically including: Step S3.3.1: Divide the input key sequence into several batches of fixed size based on the distribution characteristics of the keys; Step S3.3.2: Within each batch, the first data point is forcibly fixed as the support point of the batch to eliminate the serial dependency between points, and the slope range of all keys in the batch relative to the support point is calculated in parallel by hardware. Step S3.3.3: Update the slope range of the line segment sequentially, temporarily store it using the reprocessing queue, and prioritize the processing of the remaining subsequent keys due to the termination of the current line segment in the next batch.
4. The LSM tree-learning compression method based on computing storage devices according to claim 1, characterized in that, The method further includes: Step S5: The host receives a query request. If the query is not found in the hot data layer, the host uses the learned index model cached on the host to predict the physical address range of the target key on the computing storage device (CSD).
5. The LSM tree-learning compression method based on computing storage devices according to claim 4, characterized in that, The method further includes: Step S6: The host sends a batch of query requests containing SSTable identifiers and predicted address ranges to the computing storage device CSD.
6. The LSM tree-learning compression method based on computing storage devices according to claim 5, characterized in that, The method further includes: Step S7: The computing storage device (CSD) uses its onboard search engine to perform scanning and filtering directly within the storage medium based on the received address range, and only returns the successfully matched key-value pairs to the host.
7. An LSM tree-learning compression system based on computing storage devices, characterized in that, include: Host-side module: Used to manage the hot data layer and global metadata of the LSM tree, and responsible for generating compression tasks and processing address prediction for query requests; Computational Storage Device (CSD) Module: Connected to the host module via a bus, it stores the cold data layer of the LSM tree and integrates a programmable logic unit to perform offloaded computational tasks; Learning-based compression engine: Deployed in the computing storage device CSD module, configured to perform SSTable merge operations and parallel training of the index model; Collaborative indexing framework: configured to maintain data distribution consistency between the host module and the computing storage device (CSD) module, ensuring data flow and model updates are synchronized between storage tiers.
8. The LSM tree-learning compression system based on computing storage devices according to claim 7, characterized in that, The learning-based compression engine includes: The data loading unit is used to manage the dual buffers inside the computing storage device (CSD) to achieve overlap between reading data from flash memory and pipeline processing; The execution pipeline unit contains comparators, key-value mergers, and model training accelerators that work in parallel, enabling simultaneous data compression and model building in a single data traversal. The data writing unit is used to persistently store the generated new SSTable data and upload the generated model parameters to the host module via a high-speed bus.
9. The LSM tree-learning compression system based on computing storage devices according to claim 8, characterized in that, The model training accelerator is configured to perform the parallel piecewise linear regression (PPLR) algorithm in hardware, including: Batch processing logic unit, used to divide the key stream into independent batches and assign fixed support points; The parallel slope calculation unit utilizes the parallelism of the field-programmable gate array (FPGA) to simultaneously calculate the upper and lower bound slopes of multiple key values within a batch. The state update and rollback unit is used to detect error limits and push the remaining data into the reprocessing queue when the line segment termination is detected.
10. The LSM tree-learning compression system based on computing storage devices according to claim 7, characterized in that, The computing storage device (CSD) module also includes: An onboard search engine is configured to parse batch query requests sent by the host module. The onboard search engine utilizes the high bandwidth of the computing storage device (CSD) to directly read data blocks from the storage medium, performs in-situ scanning based on the predicted range, and filters invalid data based on key-value matching results to reduce the amount of data transfer between the host and the device.