KV separation LSM tree storage index acceleration method and device based on intelligent SSD

Abstract

The invention relates to a KV separation LSM tree storage index acceleration method and device based on an intelligent SSD, and the method comprises the steps that a KV separation LSM tree storage module is constructed at a host end, the KV separation LSM tree storage module comprises a GC manager and a GC scheduler, the GC manager constructs and manages an effective mapping table to record the validity of KV data used for each position in a value file, and the GC scheduler is used for scheduling the validity of the KV data used for each position in the value file; the GC scheduler is used for starting, scheduling and result writing-back of the GC calculation unit; a GC calculation unit is constructed at the intelligent SSD end to unload the garbage collection module from the host end to the intelligent SSD end, and the GC calculation unit carries out garbage collection processing at the intelligent SSD end in a decoding, processing and encoding mode without data dependence; and determining the parameter configuration of the deployment number of the GC computing units meeting the throughput demand. According to the method, the garbage collection operation of the KV separate storage module is unloaded based on the intelligent SSD end architecture, and the purposes of avoiding conflict between GC and read-write operation, efficiently utilizing intelligent SSD end hardware resources and cooperatively managing software and hardware states are achieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a KV separation LSM tree storage index acceleration method and device based on intelligent SSD. Background Art

[0002] In the era of big data, storage technology is experiencing unprecedented challenges. With the surge in data volume, traditional storage systems can no longer meet the growing performance requirements. Key-value storage systems based on LSM trees (Log-Structured Merge-tree) have become a hot topic of research due to their excellent write performance. Especially when processing large-value data, the key-value separated LSM tree significantly reduces the space occupied by data by storing keys and values ​​separately and replacing the actual value data with the value address index, effectively alleviating the read-write amplification problem in the data compression process. However, this architecture also faces the problem of expired data accumulation. The current garbage collection (GC) mechanism determines the validity of data by scanning value files and querying LSM trees, and then filters out invalid values ​​and writes back valid values. This process has a significant impact on system performance. When expired data accumulates to a certain extent, the frequent triggering of garbage collection will cause a sharp drop in system throughput. In addition, although adopting a conservative garbage collection strategy reduces I / O competition, it will increase the computational burden of the central processing unit (CPU), because data encoding, decoding, verification, and merging are all CPU-intensive operations.

[0003] To solve these problems, the Near Data Processing (NDP) architecture provides a new solution. By offloading computing tasks to memory, the NDP architecture can reduce CPU usage, release bus bandwidth, and reduce data movement overhead. As a commercial hardware representative of the NDP architecture, the internal computing unit of the smart SSD (Solid State Drive) can directly access data in the storage medium without going through the complex operating system storage stack and interconnection bus. However, the current garbage collection mechanism optimization work generally faces the problem that data and computing optimization cannot be achieved at the same time, and often requires a trade-off between computing overhead and read-write overhead. At present, no research work has used the near data architecture to solve the bottleneck problem of the garbage collection mechanism. By intelligently offloading tasks, it is possible to break through the limitations of host resources, request latency, and energy consumption, which is considered to be a potential solution. However, in smart SSDs, due to differences in computing environments, simple task offloading cannot directly meet the needs, and targeted processing is also required in terms of task context, resource management, data movement, etc.

[0004] In summary, the defects of the existing technology are that the garbage collection mechanism causes a sharp drop in system throughput and it is difficult to strike a balance between computing and storage optimization.

[0005] CN118550474A discloses a separate key-value pair storage system, including a host end and a storage end, the host end remotely accesses the storage end through an RDMA network; the host end is used to receive a user's read request and search for a target key-value pair from the host end, if it does not hit, the NVMe read instruction is modified, and the high 32 and low 32 bits of the key-value pair offset value are respectively filled into the NVMe read instruction and sent to the storage end; the storage end parses the TFilter metadata used to locate the target key-value pair in the NVMe read instruction, searches for the target key-value pair from the SGE, and modifies the starting address and length of the SGE according to the TFilter metadata to filter the key-value pairs outside the target key-value pair, and sends it to the host end as the SGE to be sent. In this system, since the target key-value pair is filtered out before being sent, the network traffic overhead of the transmitted data is greatly reduced, and the transmission rate is improved. However, this technical solution cannot solve the bottleneck problem of the garbage collection mechanism, that is, it cannot balance the problem between computing and storage optimization.

[0006] The present invention hopes to provide a KV separation LSM tree storage index acceleration method and device based on intelligent SSD, hoping to further reduce the impact of the garbage collection mechanism on system performance and achieve a balance between computing and storage optimization.

[0007] In addition, on the one hand, there are differences in understanding among those skilled in the art; on the other hand, the applicant studied a large number of documents and patents when making the present invention, but due to space limitations, not all details and contents are listed in detail. However, this does not mean that the present invention does not have the characteristics of these prior arts. On the contrary, the present invention already has all the characteristics of the prior art, and the applicant reserves the right to add relevant prior art to the background technology. Summary of the invention

[0008] Existing technical solutions cannot reduce the computational burden of the garbage collection mechanism on the smart SSD side. Due to the characteristics of NAND flash memory, it needs to be erased before writing, and the erasure is in blocks, which results in invalid data being generated for both data deletion and overwriting. The garbage collection process is to copy all valid data to blank pages in different data areas, erase all data units in the current data area, and then start writing new data to the data area that has just been erased. This process has a great impact on the read and write performance and life of the smart SSD side. When implementing the garbage collection steps, the smart SSD side needs to make a trade-off between computational overhead and read and write overhead. The high overhead of garbage collection will significantly reduce the performance of the smart SSD side and will also directly affect the life of the smart SSD side. The computing unit on the smart SSD side is indeed different from the traditional computing environment. What the smart SSD side needs is not a simple task transplantation, but a deep understanding of its special environment and constraints and a customized solution.

[0009] In view of the deficiencies in the prior art, the present invention provides, from a first aspect, a KV separation LSM tree storage index acceleration method based on an intelligent SSD, comprising: constructing a KV separation LSM tree storage module on a host side, the KV separation LSM tree storage module comprising a GC manager and a GC scheduler, wherein the GC manager constructs and manages a valid mapping table to record the validity of KV data used at each position in a value file, and the GC scheduler is used for starting, scheduling and writing back results of a GC computing unit; constructing a GC computing unit on an intelligent SSD side to unload a garbage collection module from the host side to the intelligent SSD side, the GC computing unit performing garbage collection processing on the intelligent SSD side in a decoding, processing and encoding manner without data dependency; and determining parameter configuration for the number of GC computing units deployed to meet throughput requirements.

[0010] The intelligent SSD end has independent computing power and can handle garbage collection tasks without interfering with host resources. Therefore, by offloading the garbage collection operation to the intelligent SSD end, the I / O and CPU resources are effectively released. By offloading the garbage collection operation to the architecture of the intelligent SSD end, the present invention can make full use of the parallel processing capability of the intelligent SSD end and reduce the I / O and CPU burden of the host end. This method greatly improves the efficiency of the garbage collection operation while reducing the occupation of system resources.

[0011] According to a preferred embodiment, the method includes: a GC calculation unit decodes the value file to be garbage collected and the valid mapping table corresponding to the value file, combines the KV data of the value file with the validity status of the valid mapping table, and generates a character stream containing the KV data of the validity status. A GC management and scheduling strategy is designed based on unloading to avoid conflicts between garbage collection operations and read and write operations, efficiently utilize hardware resources on the smart SSD end, and coordinate management of software and hardware status, giving full play to the advantages of hardware unloading.

[0012] According to a preferred embodiment, the processing steps of the GC calculation unit include: traversing the decoded character stream in the form of a data stream, determining the write / read type according to the validity status of the valid mapping table, and filtering invalid KV data.

[0013] By running garbage collection operations on the smart SSD side and combining them with scheduling strategies, we can ensure that the garbage collection process is separated from read and write operations, reduce resource competition between tasks, and thus achieve efficient resource utilization. This design strategy ensures that read and write operations will not be interrupted during the garbage collection operation, thereby improving the overall performance of the system. The resources on the smart SSD side can be used more effectively, and the coordination between software and hardware is optimized through scheduling strategies.

[0014] According to a preferred embodiment, the encoding step of the GC computing unit includes: generating a new value file from the unfiltered KV data and writing it back to the solid state drive, and rewriting the metadata into the KV separation LSM tree storage module.

[0015] The ability of the smart SSD to handle data writeback makes the garbage collection process independent of the I / O operations on the host side, thus avoiding excessive use of network and CPU resources. This approach reduces the I / O burden on the host side because GC calculations and writes are completed on the smart SSD side, without the need for frequent data transfer.

[0016] According to a preferred embodiment, the GC manager collects expiration information of KV key-value pairs during the data compression process to build or update a valid mapping table. The GC scheduler collects garbage collection data, selects the timing to wake up the hardware, and wakes up the GC computing unit on the smart SSD side, thereby completing the writeback of GC results.

[0017] The GC manager and scheduler work together to accurately determine and coordinate resource usage, reduce operational conflicts, and improve the efficiency of intelligent SSDs. The coordination between the GC manager and scheduler ensures a balance between the garbage collection process and data access, and optimizes the timing of GC activation to better utilize hardware resources.

[0018] According to a preferred embodiment, the execution steps of the GC scheduler include: calculating the number of idle GC computing units and the quota allocation data of the idle FPGA memory; reading the value file from the internal path between the solid-state drive and the field programmable gate array (FPGA) to the GC computing unit, reading the valid mapping table to the GC computing unit through the PCIe bus between the host end and the smart SSD end, starting the GC computing unit and waiting for the garbage collection process to be completed, and writing the results of the garbage collection process back to the KV separation LSM tree storage module.

[0019] The reasonable partitioning of FPGA memory and the optimization of data transmission on the PCIe bus maximize the throughput of the GC computing unit and give full play to the performance within the hardware limitations of the smart SSD end. This approach optimizes the data transmission path and resource allocation through effective field programmable gate array (FPGA) and memory management, thereby further improving the efficiency and throughput of the system.

[0020] The present invention provides a KV separation LSM tree storage index acceleration device based on an intelligent SSD from a second aspect, comprising an intelligent SSD end and a host end, wherein the intelligent SSD end is communicatively connected with the host end, and a KV separation LSM tree storage module is constructed on the host end, and the KV separation LSM tree storage module comprises a GC manager and a GC scheduler, wherein the GC manager constructs and manages a valid mapping table to record the validity of KV data used in each position in a value file, and the GC scheduler is used for starting, scheduling and writing back results of a GC computing unit; constructing a GC computing unit on the intelligent SSD end to unload a garbage collection module from the host end to the intelligent SSD end, and the GC computing unit performs garbage collection processing on the intelligent SSD end in a decoding, processing and encoding manner without data dependency; and determining parameter configuration of the number of GC computing units deployed to meet throughput requirements.

[0021] By building a GC computing unit on the smart SSD side, the garbage collection module is offloaded from the host side to the smart SSD side. With this offloading, the GC computing unit can decode, process, and encode on the smart SSD side without causing data dependency issues. The architecture of the smart SSD side solves the I / O and CPU bottlenecks in the garbage collection process. By independently performing garbage collection operations on the smart SSD side, the system effectively reduces the burden on the host side, improves the efficiency of garbage collection operations, and optimizes resource utilization.

[0022] The device of the present invention processes garbage collection tasks through an intelligent SSD end, reduces data transmission between a host and an SSD, reduces the burden of bus transmission, and improves the concurrency and efficiency of data processing.

[0023] According to a preferred embodiment, the GC calculation unit is configured to: decode the value file to be garbage collected and the valid mapping table corresponding to the value file, combine the KV data of the value file with the validity status of the valid mapping table, and generate a character stream of KV data containing the validity status.

[0024] This step ensures that the garbage collection operation can identify valid and invalid data, thereby optimizing the garbage collection process, improving system performance, and effectively reducing the storage and computing overhead caused by invalid data. By directly decoding and combining states on the smart SSD side, the number of times data needs to be transferred between the host side and the smart SSD side is reduced, reducing the latency and bandwidth requirements of data transmission.

[0025] According to a preferred embodiment, the GC calculation unit is further configured to: traverse the decoded character stream in the form of a data stream, determine the write / read type according to the validity status of the valid mapping table, and filter out invalid KV data.

[0026] By filtering invalid data, the garbage collection process reduces unnecessary data processing and storage, further improving system efficiency and performance and ensuring data validity. Failed data filtering is completed on the smart SSD side, reducing the amount of data that needs to be transmitted to the host side, thereby optimizing the data transmission path and increasing data processing speed.

[0027] According to a preferred embodiment, the GC calculation unit is further configured to: generate a new value file from the unfiltered KV data and write it back to the solid-state drive, and rewrite the metadata into the KV separation LSM tree storage module. This method ensures the integrity and consistency of the data after the garbage collection process, while maintaining the correct operation state of the system by updating the metadata. By completing data writeback and metadata updates directly inside the smart SSD end, the number of external data transmissions and the time required for data processing are reduced, further improving the response speed of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of module connection of a KV separation LSM tree storage index acceleration device based on intelligent SSD provided by the present invention;

[0029] Figure 2 It is a logical schematic diagram of the KV separation LSM tree storage index acceleration method based on intelligent SSD provided by the present invention;

[0030] Figure 3 It is a structural schematic diagram of the GC calculation unit provided by the present invention;

[0031] Figure 4 is a calculation flow chart of garbage collection task offloading provided by the present invention;

[0032] Figure 5 It is a schematic diagram of the operation logic of the GC scheduler provided by the present invention;

[0033] Figure 6 It is a schematic diagram of the operation logic of the GC manager provided by the present invention;

[0034] Figure 7 It is a flow chart of the processing process of the read and write request provided by the present invention. DETAILED DESCRIPTION

[0035] The following is a detailed description with reference to the accompanying drawings.

[0036] The present invention explains some of the noun terms.

[0037] KV separation (Key-Value Separation): is a storage optimization technology commonly used in key-value databases. It improves performance and resource utilization efficiency by storing keys and values ​​separately. Keys are usually small and frequently accessed, so they are stored in fast-access media such as memory, while values ​​are large and frequently updated, so they can be stored on slower disks. The KV separation method not only reduces memory usage, but also reduces the write amplification effect, thereby improving the overall performance and response speed of the system.

[0038] Garbage Collection (GC) in KV separation LSM tree storage is a file management module used to clean invalid data and improve storage efficiency. Its main function is to release the storage space occupied by a large number of invalid values ​​generated by data insertion, update and deletion, and reduce I / O operations to improve storage system performance. By identifying expired and invalid KV data in the value file, the garbage collection module can safely release the SSD storage space occupied by this data, thereby simplifying the developer's file management tasks and improving the robustness and stability of the program. However, the garbage collection process may have a temporary impact on system performance because in some cases it needs to suspend the normal execution of the program to ensure the consistency of memory and external storage operations.

[0039] Smart SSD (Solid State Drive): A solid-state storage device with integrated processing capabilities that can perform data processing tasks directly inside the storage medium without relying on the CPU on the host side. Smart SSDs are equipped with a dedicated hardware accelerator field programmable gate array (FPGA), which allows calculations to be performed while the data is still stored inside the device, thereby reducing data transmission delays and bandwidth consumption. Smart SSDs effectively reduce the computing burden on the host side through the Near Data Processing (NDP) architecture and optimize the overall performance of the system. They are suitable for scenarios such as big data processing, real-time analysis, and high-performance computing. Smart SSD terminals refer to electronic devices equipped with smart SSDs.

[0040] FPGA (Field-Programmable Gate Array): Field-programmable gate array is an integrated circuit that can be configured by the user on-site through a hardware description language (such as VHDL or Verilog). FPGA consists of an array of programmable logic cells that can be used to implement complex digital computing logic. Through flexible configuration, FPGA can adapt to a variety of applications, from digital signal processing and real-time data processing to custom computing accelerators. Compared with application-specific integrated circuits (ASICs), FPGAs have significant advantages in design cycle and flexibility because their functions can be reprogrammed at the hardware level.

[0041] FPGA memory: refers to the storage resources integrated inside the FPGA memory chip, which is used to support temporary data storage and fast access of logic circuits. FPGA memory chips usually include on-chip random access memory (RAM), read-only memory (ROM), block memory (Block RAM, BRAM) and distributed RAM. FPGA memory chips provide the necessary storage space for complex calculations and data processing, supporting the implementation of simple buffered data and lookup tables to more complex state machines and caches. The flexible configuration capability of this memory enhances the performance and adaptability of FPGAs when processing a variety of tasks and algorithms.

[0042] SSD controller: The core component in a smart SSD, responsible for managing and coordinating the read and write operations of data within the smart SSD. The SSD controller acts as a bridge between the host side and the NAND flash memory chip, performing tasks such as error correction, garbage collection, wear leveling, and data encryption through complex firmware algorithms. The performance and efficiency of the SSD controller directly affects the speed, reliability, and life of the SSD. It ensures the efficient layout and transmission of data between different storage blocks to optimize overall performance.

[0043] NAND chip: A non-volatile memory component used to store data in a smart SSD. NAND flash memory stores data in blocks, with high density and low cost, making it ideal for large-capacity data storage applications. It stores bits of data through an array of electronic cells. Common types include SLC (single-layer cell), MLC (multi-layer cell), TLC (triple-layer cell), and QLC (quadruple-layer cell), each of which differs in performance, durability, and cost. The characteristics of the NAND chip directly determine the storage capacity, durability, and data processing speed of the smart SSD.

[0044] GC Manager (Garbage Collection Manager): A component or module responsible for managing and executing the garbage collection process. GC managers are usually present in storage systems, especially in KV-separated LSM tree storage systems. The main task of the GC manager is to identify, recycle, and release storage space that is no longer referenced by the program so that the system can reuse these resources. The GC manager reduces redundant space occupation and the complexity of manual file management through automated garbage collection processes, thereby improving the stability and efficiency of the software.

[0045] GC Scheduler (Garbage Collection Scheduler): A component or module responsible for scheduling and optimizing the garbage collection process. In file management, the GC scheduler determines the specific timing and frequency of garbage collection to minimize the impact on application performance. The GC scheduler intelligently coordinates the execution time of garbage collection by analyzing the system's redundant space occupancy ratio, program execution status, and resource availability. An effective GC scheduler can ensure efficient use and timely recycling of storage resources while ensuring system performance.

[0046] GC Compute Unit: Generally refers to the part of the garbage collection process that is responsible for performing compute-intensive operations. These operations may include identifying expired data, organizing and compressing storage space, and updating references in memory. The purpose of the GC Compute Unit is to perform garbage collection operations efficiently and minimize its impact on application performance. By optimizing these compute-intensive tasks, the GC Compute Unit can help improve the efficiency of the garbage collection process, thereby improving the performance and responsiveness of the entire system.

[0047] Host side: includes central processing unit CPU and memory DRAM. The program running the KV separation LSM tree storage module runs in the CPU and memory DRAM of the host side. The central processing unit CPU handles all system computing operations, such as reading and writing key-value pairs, data compression (Compaction), file reading and writing, and GC. System data structures, such as memory tables, are cached in the memory. Preferably, the host side can be a server and a server group. Preferably, the host side can also be a dedicated integrated chip with the same computing power and operating capabilities.

[0048] ValidMap: is a data structure used to identify which data items or memory areas are valid. It can be applied to multiple scenarios. For example, in memory management, ValidMap is used to mark which memory addresses are already allocated and available; in garbage collection mechanisms, it is used to identify which objects are still referenced and need to be retained rather than recycled; in data caching systems, ValidMap helps determine which cached data is the latest or valid to avoid using expired information. Different systems may have specific implementations and uses of ValidMap, but its core function is to represent the validity status of data through a mapping structure.

[0049] Example 1

[0050] The present invention provides a KV separation LSM tree storage index acceleration device based on intelligent SSD, including a host end and an intelligent SSD end. Figure 1 As shown, the smart SSD end and the host end are connected by a PCIe (Peripheral Component Interconnect Express) bus. The PCIe bus has the characteristics of high bandwidth and low latency, which can meet the needs of fast data transmission between the smart SSD end and the host end. Specifically, the smart SSD on the smart SSD end is connected to the mainboard of the host end through a PCIe slot. The PCIe slot provides electrical connection and physical fixation to ensure stable communication between the smart SSD end and the host end. The PCIe communication protocol is used for data transmission between the smart SSD end and the host end. The PCIe protocol defines the format, transmission method and error handling mechanism of the data packet to ensure accurate data transmission.

[0051] Preferably, if Figure 1 and Figure 2 As shown in FIG. 1 , the smart SSD includes a solid state drive (SSD) and a field programmable gate array (FPGA). The solid state drive (SSD) provides a large-capacity data storage function. Figure 1As shown, the solid state drive (SSD) includes an SSD controller and a NAND chip as a storage medium. The SSD controller is connected to the NAND chip so that the SSD controller manages the read and write operations of the data. The field programmable gate array (FPGA) is connected and communicates with the solid state drive (SSD) through internal lines, and works together to improve data processing efficiency. In the smart SSD, the field programmable gate array (FPGA) can implement various functions, such as data processing, encryption, compression, etc. Specifically, the solid state drive (SSD) is responsible for the read and write operations of the storage data, and the field programmable gate array (FPGA) is used for real-time processing and calculation of the data. Drivers for the SSD driver and the FPGA driver are also provided in the solid state drive (SSD).

[0052] The smart SSD is equipped with a PCIe interface. The solid state drive (SSD) and the field programmable gate array (FPGA) are connected to the PCIe interface of the smart SSD respectively. This connection allows high-speed data transmission to meet the computing requests of the solid state drive (SSD) and the field programmable gate array (FPGA). The solid state drive (SSD) and the field programmable gate array (FPGA) are mapped to different PCIe address spaces (such as Figure 1 This means that in the system's PCIe architecture, they each occupy an independent address range and can handle the read and write requests of the solid-state drive (SSD) and the computing requests of the field programmable gate array (FPGA) respectively.

[0053] Preferably, when the host sends a read or write request to the smart SSD, the host sends request data including target address, data length and other information to the smart SSD through the corresponding PCIe address. After receiving the request data through the PCIeSwich (PCIe switch), the SSD controller of the smart SSD parses and determines the content of the request data and performs corresponding operations according to the request type. If it is a read request, the SSD controller reads data from the solid-state drive (SSD) and sends the data to the host through the PCIe bus. If it is a write request, the SSD controller writes the data sent by the host into the NAND chip and sends a response signal to the host after the write is completed.

[0054] like Figure 1As shown, when the host sends a read / write request to the smart SSD, the host passes the calculation request to the SSD device driver (SSD driver for short). The SSD device driver will encapsulate the control information of the calculation request to make it conform to the format and protocol for communication with the smart SSD. When the SSD controller of the smart SSD receives the calculation / read / write request, it will allocate and schedule tasks according to the type and priority of the calculation request, and then use the programmable logic circuit of the corresponding field programmable gate array (FPGA) inside it to perform calculations according to the instructions and data in the calculation request. The field programmable gate array (FPGA) is provided with an FPGA memory chip.

[0055] like Figure 2 As shown, the overall device of the present invention is divided into a host end and a smart SSD end.

[0056] The host side is configured with a memory table and an immutable memory table of a KV separation LSM tree storage module, as well as a GC manager and a GC scheduler proposed by the present invention. The memory table is responsible for caching the write request KV data received by the overall device, and when it is full, it is converted into an immutable memory table and refreshed and written to the solid-state drive (SSD) storage of the intelligent SSD side. The memory table is also used for reading KV data. The GC manager is responsible for collecting and maintaining the valid mapping table (ValidMap) structure from the results of data compression (Compaction) of the SS table. Data compression (Compaction) refers to the process of merging multiple small SS tables into a larger SS table to optimize query performance and storage efficiency. The immutable memory table is continuously refreshed. The GC scheduler is responsible for the execution of garbage collection operations, reading the corresponding valid mapping table (ValidMap) from the GC manager and transmitting it to the GC computing unit of the FPGA, synchronously transmitting the value file to the GC computing unit through the I / O control unit, and then starting the processing process of the GC computing unit through the computing control unit, and finally rewriting the metadata of the processing result into the memory table. Preferably, the FPGA has FPGA computing power. FPGA computing power refers to the computing power of a field programmable gate array (FPGA).

[0057] The smart SSD end includes a solid-state drive (SSD) and a field-programmable gate array (FPGA). The solid-state drive (SSD) stores key files (SS tables) and value files. The GC manager on the host side will continuously perform data compression (Compaction) on the key files and garbage collection on the value files to organize the data to maintain the stability of the storage space. Several GC computing units are deployed in the field-programmable gate array (FPGA). The GC computing unit is used to offload the garbage collection operation originally performed on the host side in the KV separation LSM tree storage. Under the control of the I / O control unit of the GC scheduler on the host side, the memory data of the field-programmable gate array (FPGA) and the file storage data of the solid-state drive (SSD) can be directly transmitted through the internal bus channel of the smart SSD end.

[0058] This configuration enables the smart SSD to have garbage collection computing capabilities, and then unloads the garbage collection program from the KV separation LSM tree storage module, extracts the garbage collection task from the context of the host, completes the file transmission preparation through the internal path of the smart SSD, and completes the garbage collection calculation and writes back the corresponding results (metadata) on the smart SSD. This process is completed with the assistance and control of the GC manager and GC scheduler on the host.

[0059] The steps for the KV separation LSM tree storage module on the intelligent SSD side to process read and write requests are as follows: Figure 7 shown.

[0060] S000: Start.

[0061] S101: When the device composed of the host end and the intelligent SSD end is started, the KV separation LSM tree storage module configures parameters to create new related resources. In particular, the KV separation LSM tree storage module evokes the required field programmable gate array (FPGA) and GC computing unit instances on the intelligent SSD end according to the number of currently idle GC computing units.

[0062] S102: Initiate a request.

[0063] After the startup is complete, the KV separation LSM tree storage module can receive a request initiated by an external system or user. The request can be a read operation or a write operation.

[0064] S103: Determine the type.

[0065] The KV separation LSM tree storage module determines the type according to the request, that is, determines the write type and the read type. If the request is of the read type, S104 is executed; if the request is of the write type, S109 is executed.

[0066] S104: When the request is of a read type, the KV separation LSM tree storage module performs a memory table search (MemTable), that is, searches for the required data in the memory table.

[0067] S105: Determine whether the required data is found. If the required data is found, the data is returned and the process ends.

[0068] S106: If the required data is not found, the KV separation LSM tree storage module performs a traversal search of the SS table (SSTable).

[0069] S107: When the required data is found in the SS table (SSTable), the data is returned and the process ends.

[0070] S108: When the required data is not found in the SS table (SSTable), a not found result is returned and the query ends.

[0071] S109: When the request is of a write type, the KV separation LSM tree storage module writes the data into a memory table (MemTable).

[0072] S110: Return a message indicating that the writing is successful.

[0073] S111: Check whether the memory table (MemTable) is full. In other words, check whether the capacity of the memory table has reached the upper limit. If so, execute S112; if not, execute S113 (i.e., write is completed).

[0074] S112: If so, trigger the Flush operation, write the data from the memory table to the disk in the SS table (SSTable) file format (Level_0 layer in LSM), and create a valid mapping table (ValidMap) corresponding to the new SS table (SSTable), and execute S114.

[0075] S113: Writing is completed.

[0076] S114: Check whether Level_0 is full.

[0077] After the flush operation is completed, it is continued to check whether the capacity of the Level_0 layer reaches the threshold. If so, execute S115; if not, execute S113.

[0078] S115: If yes, trigger data compression (Compaction).

[0079] S116: Update the valid mapping table (ValidMap).

[0080] After data compression (Compaction) is completed, the GC manager will update the valid mapping table (ValidMap) to mark the current validity status of the data.

[0081] S117: The GC manager checks whether the garbage rate of the value file reaches a threshold value (garbage rate threshold value). If so, execute S118; if not, execute S113.

[0082] S118: If yes, create a garbage collection task (GC task) and publish it to the GC scheduler.

[0083] S119: Transmit the value file and the corresponding valid mapping table (ValidMap).

[0084] When the GC scheduler receives a new garbage collection task, the calculation control unit creates a hardware GC offload task context, and the I / O control unit transmits all value files and the corresponding valid mapping table (ValidMap) to the GC calculation unit.

[0085] S120: Trigger GC (GC calculation unit).

[0086] That is, the GC calculation unit starts the GC calculation process and waits for the completion result.

[0087] S121: Writing back metadata.

[0088] When the GC calculation unit completes the task, the metadata write-back unit in the GC manager will receive the returned GC metadata, and then write all the metadata back to the KV separation LSM tree storage module through batch write requests.

[0089] Preferably, the process of unloading the garbage collection task by the GC calculation unit includes three stages: input decoding, data calculation and output encoding.

[0090] Input decoding stage: The GC calculation unit is used to process the value files to be garbage collected and the valid mapping table (ValidMap) corresponding to each value file. Figure 3 As shown, an FPGA DRAM in the FPGA memory chip reads the value file that needs to be processed in this garbage collection task. The value file includes value file 0, value file 1, value file 2, ... and the valid mapping table (ValidMap) input from the host. The key-value pairs inside the value file are adjacent to each other. The GC calculation unit determines the offset limit by the data size, and the valid mapping table (ValidMap) is encoded with standardized character data to efficiently obtain the flag information of the corresponding position. On this basis, the GC calculation unit performs a high-parallelism pipeline decoding process on the target file.

[0091] Data calculation stage: The GC calculation unit traverses the character stream in the form of a data stream based on the valid mapping table (ValidMap) for filtering, and generates a signal for filtering or retention. During the loop iteration process, the GC calculation unit captures the decoded KV data at the current offset position and the signal generated by filtering. If the signal is to retain, the KV data is buffered and the metadata of the KV data (the size and offset position of the current KV data) is collected. If the signal is to filter, because the expired data does not need to be copied and migrated again, it will directly skip after capturing the KV data and loop to the processing of the next data.

[0092] Output encoding: The GC calculation unit re-encodes all buffered KV data in the original order to generate value file 3, that is, fills all KV data in the format of the value file and sets the file header, and then directly writes value file 3 back to the solid-state drive (SSD) through the internal bus of the smart SSD. Metadata collection is performed during the data calculation phase, that is, the metadata of each KV data is collected, and then other metadata, including the number of KVs, file size, new and old addresses and sizes of KV data, are calculated and attached to generate all metadata completed by this garbage collection task, and then transmitted to the metadata write-back unit of the host-side GC scheduler through the PCIe bus.

[0093] like Figure 4 As shown, the steps of offloading the garbage collection task by the GC computing unit are as follows.

[0094] S000: Start.

[0095] S210: In the initialization phase, the calculation unit (GC calculation unit) is initialized, the value file and the corresponding valid mapping table (ValidMap) are input, and the file header is parsed.

[0096] Specifically, the memory space is initialized, the internal communication library and modules of the smart SSD are loaded, and the GC calculation unit is ensured to be in a ready state.

[0097] The GC calculation unit receives the value file and the corresponding valid mapping table (ValidMap) input by the garbage collection task. The GC calculation unit parses the file header of the value file and extracts key information in the file header. Key information such as file format version, compression format, file size, etc., so as to correctly process the data in the file later.

[0098] S220: In the processing stage, each key-value pair data is processed cyclically, the key-value pair format is parsed, and the corresponding bit state of the valid mapping table (ValidMap) is read. Specifically, the GC calculation unit processes each key-value pair data cyclically.

[0099] For each key-value pair, first, parse the key-value pair format and determine the length of the key and value in order to correctly read and process the data.

[0100] Then, according to the position of the current key-value pair, the corresponding bit state is read from the valid mapping table (ValidMap) to determine whether the current key-value pair data is valid.

[0101] If not, that is, the key-value pair data is invalid, it is directly ignored and skipped, and the next key-value pair processing cycle is entered. If yes, that is, the key-value pair data is valid, the key-value pair data is buffered, the cycle ends, and waits for subsequent encoding processing.

[0102] In the whole process, when processing key-value data, pipeline loop and optimization technology can be used for parallel processing to improve the efficiency of data processing. For example, in the steps of parsing the key-value format, reading the corresponding bit status of the valid mapping table (ValidMap) and judging the validity, multiple key-value pairs can be processed at the same time to increase the processing throughput.

[0103] S230: In the end stage, a new value file is constructed in the KV data buffer, and the new value file is written to the solid state drive (SSD), and the metadata is written back to the host side; the computing unit ends.

[0104] Specifically, the GC calculation unit reads all key-value pair data from the KV data buffer and reconstructs a new value file according to the value file format, and writes the new value file to the solid state drive (SSD) through an internal channel.

[0105] After completing the processing of all key-value pairs, the GC calculation unit writes the metadata (such as the new address of the migrated KV data, basic information of the new value file, etc.) back to the metadata write-back unit of the GC scheduler on the host. After completing all data processing and write-back operations, the GC calculation unit is closed and cleaned up to release occupied resources, such as memory and file handles.

[0106] S240: End.

[0107] like Figure 5 As shown, the execution process of the GC scheduler includes the following steps.

[0108] like Figure 5As shown, the GC scheduler includes an I / O control unit, a computing control unit, and a metadata write-back unit. The I / O control unit controls the field programmable gate array (FPGA) to prepare and write back GC data using the internal path between the solid state drive (SSD) and the field programmable gate array (FPGA). The computing control unit connects data and controls the flow with the GC computing unit in the field programmable gate array (FPGA). The metadata generated by the GC computing unit is input into the metadata write-back unit, and then written into the host-side memory table, and then uniformly persisted to the solid state drive (SSD) storage with the flush operation of the host.

[0109] (1) The I / O control unit is responsible for data preparation and result writing back under the premise of idle GC computing units and idle DRAM. Specifically, the GC computing unit needs to input a value file and a valid mapping table (ValidMap) to perform a garbage collection task, and output a new value file and metadata. Under the control of the I / O control unit, the reading and writing back of the value file are completed by the field programmable gate array (FPGA) through the internal data path of the intelligent SSD end. The valid mapping table (ValidMap) and write-back metadata are completed through the system bus path between the host end and the intelligent SSD end. In addition, in terms of implementation, the I / O control unit uses hardware asynchronous instruction allocation and transmission to enhance the read and write parallelism between paths and tasks while avoiding concurrency errors.

[0110] (2) The computing control unit is responsible for scheduling the GC computing units. The number of computing units in the hardware is determined by the GC parameters specified when the system is running. The usage of the FPGA memory chip can be allocated and managed based on the number of computing units in use and the FPGA DRAM quota of each computing unit. The theoretical minimum memory size required for each computing unit includes: input files, reserved output files, and temporary areas for intermediate calculation values. In addition, the computing control unit is also responsible for starting and ending the GC computing unit, and this process is implemented in an asynchronous waiting manner. When the GC computing unit is executed, a callback function is started to pass the result data to the metadata writeback unit;

[0111] (3) The metadata write-back unit is responsible for writing the results of the GC calculation unit back to the KV separation LSM tree storage module, including: the basic information of the new value file (file name, key range, size, data volume) and the new address of the migrated KV data. This information is collected during the execution of the GC calculation unit and finally transmitted to the metadata write-back unit.

[0112] The GC manager includes a valid mapping table (ValidMap) maintenance submodule and a GC information collection submodule.

[0113] like Figure 6As shown in the figure, the valid mapping table (ValidMap) uses a bitmap as the basic structure and indicates the validity of each offset point data with a single bit. One value file corresponds to one valid mapping table (ValidMap). The memory space of all valid mapping tables (ValidMap) is uniformly registered or destroyed by the host-side dynamic memory management interface. The valid mapping table (ValidMap) maintenance submodule is responsible for the maintenance and reading of the valid mapping table (ValidMap) in the memory. By monitoring the validity ratio in the valid mapping table (ValidMap) corresponding to each value file, when the garbage rate threshold is exceeded, the GC manager publishes the garbage collection task in the metadata queue of the garbage collection task and assigns the task to the GC scheduler for subsequent processing. In addition, in the process of passing the garbage collection task to the GC scheduler, the valid mapping table (ValidMap) corresponding to the value file to be garbage collected will also be read and transmitted to the GC scheduler through transposition or metadata assistance. Metadata assistance means that in the entire garbage collection and validity management process, metadata provides auxiliary information, enabling the system to more effectively identify which data is invalid and needs to be recycled, and which data is still valid. Transposition refers to the rearrangement or relocation of data or metadata in memory during memory management. The purpose of this is to optimize memory usage, improve access efficiency or organize data.

[0114] Data compression: update. The GC information collection submodule collects the location information of all filtered data and passes it back at the end of each data compression (Compaction). When the collection data of the relevant positions of the invalid data is obtained, the GC manager starts the update process and traverses the invalid data. Preferably, the GC manager rebuilds each temporary valid mapping table (ValidMap) according to the file number and offset sequence in the invalid data, and finally performs a bitwise OR operation on the temporary valid mapping table (ValidMap) and the corresponding valid mapping table (ValidMap) in the global space to complete the update of the data structure.

[0115] Metadata writeback. After the garbage collection operation is completed, the files deleted by the garbage collection process will be passed to the GC manager. After receiving the metadata of the file name, the GC manager will delete the valid mapping table (ValidMap) corresponding to these files.

[0116] This method uses systematic parameter configuration to maximize the efficiency of the field programmable gate array (FPGA) hardware resources on the smart SSD side. Since the resources on the smart SSD side have an upper limit and cannot support the deployment of too many computing units, and the DRAM resources on the smart SSD side need to support the various data requirements of each GC computing unit, that is, when increasing the number of computing units to improve the overall throughput, it is necessary to consider the resource restrictions on deployment. Since the computing performance of the GC computing unit can be determined at compile time, and then by real-time monitoring of the throughput status of the entire device in processing external requests, and then theoretical modeling and solving the throughput performance of the GC computing unit, it can provide guidance for GC parameter settings under various read-write mixed scenario targets.

[0117] S310: In order to solve the GC related parameters, firstly, the current workflow characteristics and hardware resource quota are sorted out to provide necessary data for the following calculation steps.

[0118] Specifically, workflow characteristics refer to the total throughput and request type ratio of the overall device required to process external read and write requests; hardware resource quota refers to the resource quota of FPGA computing power in the smart SSD end (including CLB, FF, LUT) and the number of resources required for the deployment of each GC computing unit.

[0119] S320: The triggering condition of the garbage collection task is that the file garbage rate is greater than a threshold. Therefore, when the overall device reaches a stable stage (that is, the proportion of redundant space in the value file remains stable while the overall device continues to receive read and write requests), in order to ensure stable storage space utilization, the following formula needs to be satisfied:

[0120]

[0121] Among them, Th update Indicates a key-value pair update or delete request for the entire device. i represents the throughput of each GC computing unit, N CU Indicates the number of deployed GC computing units, and Gr indicates the garbage rate threshold triggered by GC. The system throughput before garbage collection is the theoretical upper limit of the current configuration. The GC computing unit unloaded by the hardware will not cause too much burden on the system. When the total throughput of garbage collection by each hardware unit reaches the system garbage generation rate, it can ensure that the redundant space of the entire storage is stable without damaging the system performance.

[0122] For example, take the default value of Gr as 0.5 (indicating that when the proportion of invalid data in a file is greater than 50%, garbage collection is required). Assuming that the update request throughput of the entire device is 30,000 per second and the processing throughput of the GC computing unit is 15,000 per second, the minimum number of GC computing units that need to be deployed is 4 according to the above formula.

[0123] S330: The part to be solved in the above formula is the theoretical throughput calculation of each computing unit. The calculation method is: the total time of a garbage collection task includes: the completion time of reading data, computing execution, writing data and metadata writing back.

[0124] The throughput of the GC computing unit is the quotient of the total number of data processed in a task and the total time consumed, which satisfies the following formula:

[0125]

[0126] Among them, Th i N represents the theoretical throughput of each GC computing unit. sum Indicates the total number of data processed in a garbage collection task (GC task), T read , T compute , T write and T meta They represent the completion time of reading data, executing calculations, writing data, and writing back metadata in a garbage collection task.

[0127] S340: The hardware resources such as CLB (Configurable Logic Blocks), FF (FlipFlop), and LUT (Lookup Table) required by each GC computing unit are determined during compilation. When the Vitis module is used to compile and deploy the hardware of the GC computing unit, the resource usage report automatically generated by the Vitis module records the usage of all hardware resources. Based on the number of hardware resources required by the GC computing unit, the upper limit of the number of GC computing units that can be deployed by a single smart SSD in the smart SSD end can be determined, that is, the following formula is satisfied:

[0128]

[0129] Among them, R CLB , R FF , R LUT Indicates the CLB, FF, and LUT hardware resource quota of the FPGA computing power in a single smart SSD, N CLB 、N FF 、N LUT Indicates the number of CLB, FF, and LUT resources required by a single GC computing unit and the actual number of GC computing units deployed N CU The hardware resource limit must be met, that is, the resources used cannot exceed the upper limit of the number of resources provided by the smart SSD.

[0130] S350: Based on the storage space utilization stability condition of step S320, combined with the theoretical throughput calculation method of the GC computing unit of step S330 and the resource restriction condition of S340, the storage space redundancy Gr and the number of GC computing units deployed N that meet the overall device read and write throughput requirements can be finally solved. CU Parameter configuration to run with corresponding parameters when the KV separation LSM tree storage module is started.

[0131] For example, the input KV data volume of a garbage collection task is 300,000, and the completion time of reading data, computing execution, writing data and metadata writing back is 2 seconds, 12 seconds, 2 seconds and 4 seconds respectively, that is, the total time of the garbage collection task is 20 seconds, and the processing throughput of the GC computing unit can be calculated to be 15,000 / second. For example, the number of CLBs, FFs and LUTs of the field programmable gate array (FPGA) in the smart SSD is 70,000, 500,000 and 1,000,000 respectively, and the number of resources required for the deployment of a single computing unit is 8,000, 20,000 and 30,000 respectively, which can be solved to get N cu The maximum number is 8; the throughput of the overall device for processing update requests is 30,000 per second, and it can be solved that the minimum number of GC computing units that need to be deployed is 4 (less than the upper limit of 8).

[0132] Example 2

[0133] This embodiment is a further improvement on Embodiment 1, and the repeated contents will not be repeated here.

[0134] For example, for large e-commerce platforms, data is updated and written very frequently, and the data storage structure is updated and changed in real time (such as updating and writing product information, user behavior log records, transaction order data management, etc.). In such an environment, the performance of the storage engine determines whether the platform can provide high-quality and stable services.

[0135] The method and device of the present invention can be applied to e-commerce platforms to optimize the data processing efficiency thereof. The specific steps are as follows:

[0136] (1) KV Separation LSM Tree Storage

[0137] In e-commerce platforms, large quantities of data are stored in the form of key-value pairs in KV-separated LSM tree storage modules. Taking user behavior log records as an example, user behavior log records play a vital role in analyzing user behavior, optimizing user experience, and conducting precision marketing. When users perform various operations on e-commerce platforms, such as browsing products, searching products, adding shopping carts, and placing orders, the system will generate user behavior logs in real time. These logs contain the user's unique identifier (such as user ID), behavior type (such as browsing, searching, adding shopping carts, etc.), the time when the behavior occurred, and information about related products. Using the KV-separated LSM tree storage module, the key of a log data can be the unique identifier of the user behavior, such as a string consisting of user ID, behavior type, and timestamp, and the value contains specific behavior details, such as product ID, search keywords, etc. In the KV-separated storage method, the key will be stored in the index file of the KV-separated LSM tree storage module, and the value will be stored in another data file area. The advantage of this is that the storage location of the value can be quickly located according to the key index when querying, improving query and access efficiency.

[0138] (2) Acceleration method based on intelligent SSD

[0139] In e-commerce platforms, a large amount of data in the form of key-value pairs, such as product information, user data, and order records, needs to be written and updated in real time. For example, when user A browses, searches, or trades, or when product B is removed from the shelves or its configuration is modified, the original key-value data will become expired, and garbage collection needs to be performed continuously to process expired data in a timely manner. Using the KV-separated LSM tree storage module built on the intelligent SSD end, the GC computing unit on the intelligent SSD end solves the problem of expired data accumulation in the value area non-invasively, avoiding the impact on the system's read and write processing performance due to the need to efficiently process expired data.

[0140] After the method and device of the present invention are applied to the e-commerce platform, the e-commerce platform has the following advantages:

[0141] First, solve the I / O and CPU bottleneck problems: offload garbage collection operations from the host side to the smart SSD side, use its high-speed in-memory computing power and high-bandwidth internal bus characteristics to reduce system I / O operations, avoid CPU bottlenecks caused by garbage collection, and ensure that the e-commerce platform can still maintain stable performance in high-concurrency data reading and writing scenarios (such as a large number of users placing orders and querying product information during promotional activities).

[0142] Second, improve computing performance: The GC process of the hardware GC computing unit in the smart SSD is divided into three stages: decoding, calculation, and encoding, and highly parallel pipeline optimization is performed. Compared with the traditional CPU processing method, it greatly reduces the calculation time of operations such as data encoding and decoding, verification and merging, and improves the overall system performance.

[0143] Third, efficient management of hardware resources: Through systematic parameter configuration, the system is provided with optimal GC parameter recommendations based on the workflow characteristics, hardware resource quotas, and target performance requirements of the e-commerce platform. For example, the trigger threshold of garbage collection, the number of computing units, and other parameters are reasonably determined to ensure that the redundant storage space is stable without affecting system performance, while efficiently utilizing the hardware resources of the smart SSD end.

[0144] Fourth, optimize resource utilization efficiency: The I / O control unit in the GC scheduler uses the internal data path of the smart SSD to achieve parallel reading and writing to avoid concurrency errors; the GC computing unit reasonably allocates the number of computing units and FPGA DRAM quota to improve resource utilization; the metadata write back unit coordinates the write back of GC results through the buffer queue and batch flushing mechanism to reduce the impact on performance, so that the hardware resources of the smart SSD can be fully utilized in the data storage of the e-commerce platform.

[0145] It should be noted that the above-mentioned specific embodiments are exemplary, and those skilled in the art can come up with various solutions inspired by the disclosure of the present invention, and these solutions also belong to the disclosure scope of the present invention and fall within the protection scope of the present invention. Those skilled in the art should understand that the present invention specification and its drawings are illustrative and do not constitute a limitation on the claims. The scope of protection of the present invention is defined by the claims and their equivalents. The present invention specification contains multiple inventive concepts, such as "preferably" and "according to a preferred embodiment", which means that the corresponding paragraph discloses an independent concept, and the applicant reserves the right to file a divisional application based on each inventive concept.

Claims

1. A KV separation LSM tree storage index acceleration method based on intelligent SSD, characterized in that: include: A KV separation LSM tree storage module is constructed on the host side, wherein the KV separation LSM tree storage module includes a GC manager and a GC scheduler. The GC manager constructs and manages a valid mapping table to record the validity of KV data used at each location in the value file. The GC scheduler is used for starting, scheduling and writing back the results of the GC computing unit; A GC calculation unit is constructed on the smart SSD end to offload the garbage collection module from the host end to the smart SSD end, and the GC calculation unit performs garbage collection processing on the smart SSD end in a decoding, processing and encoding manner without data dependency on each other; Determine the parameter configuration for the number of GC computing units deployed to meet the throughput requirements.

2. The method according to claim 1, characterized in that The method comprises: a GC calculation unit decodes a value file to be garbage collected and a valid mapping table corresponding to the value file; The KV data of the value file is combined with the validity status of the validity mapping table to generate a character stream of the KV data including the validity status.

3. The method according to claim 1 or 2, characterized in that: The processing steps of the GC calculation unit include: The decoded character stream is traversed in the form of a data stream, the write / read type is determined according to the validity status of the valid mapping table, and invalid KV data is filtered.

4. The method according to any one of claims 1 to 3, characterized in that: The encoding step of the GC calculation unit includes: The unfiltered KV data is used to generate a new value file and written back to the solid-state drive, and the metadata is rewritten into the KV separation LSM tree storage module.

5. The method according to any one of claims 1 to 4, characterized in that: The GC manager collects the expiration information of KV key-value pairs during the data compression process to build or update the valid mapping table. The GC scheduler collects garbage collection data, selects a hardware wake-up opportunity, and wakes up the GC calculation unit on the smart SSD side, thereby completing the GC result write-back.

6. The method according to any one of claims 1 to 5, characterized in that: The execution steps of the GC scheduler include: Calculate the number of idle GC computing units and the quota allocation data of idle FPGA memory; The value file is read from the internal path between the solid state drive and the field programmable gate array (FPGA) to the GC calculation unit, and the valid mapping table is read to the GC calculation unit through the PCIe bus between the host end and the smart SSD end, Start the GC calculation unit and wait for the garbage collection process to complete, The results of the garbage collection process are written back to the KV separation LSM tree storage module.

7. A KV separation LSM tree storage index acceleration device based on intelligent SSD, characterized in that: It includes a smart SSD end and a host end, wherein the smart SSD end is connected to the host end for communication. The host end is constructed with a KV separation LSM tree storage module, and the KV separation LSM tree storage module includes a GC manager and a GC scheduler. The GC manager constructs and manages a valid mapping table to record the validity of KV data used at each location in the value file. The GC scheduler is used for starting, scheduling and writing back the results of the GC computing unit; A GC calculation unit is constructed on the smart SSD end to offload the garbage collection module from the host end to the smart SSD end, and the GC calculation unit performs garbage collection processing on the smart SSD end in a decoding, processing and encoding manner without data dependency on each other; Determine the parameter configuration for the number of GC computing units deployed to meet the throughput requirements.

8. The device according to claim 7, characterized in that The GC calculation unit is configured to: Decoding the value file to be garbage collected and the valid mapping table corresponding to the value file, The KV data of the value file is combined with the validity status of the validity mapping table to generate a character stream of the KV data including the validity status.

9. The device according to claim 8, characterized in that The GC calculation unit is further configured to: The decoded character stream is traversed in the form of a data stream, the write / read type is determined according to the validity status of the valid mapping table, and invalid KV data is filtered.

10. The device according to any one of claims 7 to 9, characterized in that: The GC calculation unit is further configured to: The unfiltered KV data is used to generate a new value file and written back to the solid-state drive, and the metadata is rewritten into the KV separation LSM tree storage module.

Images