A parallel computing method for FPGA acceleration of a Key-Value storage system

Abstract

The application discloses a kind of parallel computing methods of FPGA acceleration for Key-Value storage system, method includes: triggering compaction operation, the SSTable of key range repetition is transferred to FPGA memory;Using FPGA parallel decoding and sorting these SSTable, obtain ordered and non-repeated key-value pair set;Using FPGA parallel coding this set, form new SSTable, and these SSTable are directly written from FPGA memory to SSD, bypass CPU, and the invalid key-value pair in log file is recycled by parallel garbage collection operation.The application parallel computing method of FPGA acceleration for Key-Value storage system, using key-value separation method reduces the write amplification problem of key-value storage, and provides comprehensive optimization for the calculation load and data transmission delay in key-value storage, reduces the pressure of CPU.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optimizing Key-Value storage system compaction operation, and particularly relates to a parallel computing method for FPGA acceleration of Key-Value storage system. BACKGROUND

[0002] The rapid development of information technology has led to a significant increase in global data, with the majority being unstructured data. According to the IDC DataSphere forecast, by 2027, unstructured data will account for approximately 86.8% of global data, or about 246.9 ZB. These unstructured data come from diverse sources such as social media, generative artificial intelligence, multimedia, and have become crucial in various industries. Therefore, managing such a large and growing amount of unstructured data and building an efficient and user-friendly data storage platform remains a challenge. For the management of unstructured data, although they can be stored in traditional relational database systems, they show a bottleneck in query performance in handling frequent read-write operations of a large number of small files, and have limited optimization space. However, non-relational database systems, such as Key-Value storage (KV store) based on Log-Structured Merge-tree (LSM-tree) structure, are popular solutions for managing large-scale unstructured data. For example, LevelDB and RocksDB are applied in key-value storage to achieve effective data management of unstructured data.

[0003] KV store provides high read-write efficiency and scalability. The high performance of KV store mainly comes from the LSM-tree structure, which improves the write performance of KV store and the space efficiency on disk. In addition, its hierarchical architecture enhances the performance of KV store. In order to achieve the high write performance of LSM-tree and the efficient disk space utilization, KV store needs to perform compaction operation to periodically merge data and delete repeated data on disk. However, compaction is a computationally intensive task that requires a large amount of CPU computing resources and disk I / O, which can cause high CPU computing load and data transfer delay between disk and CPU, and thus reduce the performance of KV store, and the compaction operation can become a bottleneck of KV store. With the increase in the amount of unstructured data, compaction optimization methods become crucial. In addition, compaction will select repeated files for processing, which can cause a serious write amplification problem.

[0004] Existing methods mainly focus on optimizing the compaction process by improving the LSM-tree structure and compaction mechanism, storage media, task offloading scheme, and heterogeneous acceleration, to alleviate the performance bottleneck of compaction operation. In order to achieve parallel processing and reduce CPU load, some methods propose a collaborative computing scheme, which offloads the compaction task to collaborative computing devices (such as General-Purpose Graphics Processing Unit (GPGPU), Field-Programmable Gate Array (FPGA), Data Processing Unit (DPU), and Near-Data Processing (NDP) devices) to improve the performance of KV store. In order to reduce the write amplification problem caused by the compaction process, some methods propose a key-value separation strategy, which separates keys and values, and stores keys and value pointers in the LSM-tree, while values are stored in log files. By reducing the amount of data processed during the compaction process, write amplification is reduced to improve the performance of KV store.

[0005] In existing FPGA-accelerated compaction optimization schemes, compute-intensive tasks are generally offloaded to FPGAs to alleviate the computational load of CPUs, thereby improving computational efficiency. In addition, KV stores are integrated with FPGAs, and PCIe Peer to Peer (P2P) communication is used to reduce CPU resource usage and reduce data transfer latency.

[0006] However, these methods are difficult to optimize both computational load and data transfer latency during the compaction process. In addition, existing researches mostly focus on computational optimization, and relatively less on data transfer latency optimization. KV store, as a compute-intensive application, requires a large amount of computational resources and frequent data transfer. Therefore, it is crucial to improve the performance of KV store from both computational load and data transfer latency perspectives in the FPGA-accelerated framework.

[0007] In a key-value separation-based KV store, although the key-value separation strategy reduces the amount of data processing and moving, the compaction process still needs to traverse all the key-value pairs of SSTable and process them, such as decoding, marking and encoding, and cannot completely solve the problem of high CPU and I / O resource occupation in the compaction operation. In addition, in the key-value separation strategy, the log file is used to manage the value. In order to recycle the invalid key-value pairs (i.e. expired key-value pairs) deleted in the compaction process, garbage collection needs to be triggered regularly to release the disk space. However, the garbage collection mechanism needs to traverse the log file, retain valid key-value pairs and skip invalid key-value pairs according to the validity of each key-value pair in the log file, and continue to traverse the next key-value pair, and finally write the valid key-value pairs back to the LSM-tree, thereby completing the garbage collection task. This traversal process further increases the CPU burden, thereby causing a system performance bottleneck.

[0008] The application number 202210293809.2 discloses a key-value storage system and a garbage collection method. The application scheme directly uses persistent memory to replace disk storage data, utilizes the characteristics of large capacity and high speed of persistent memory, and constructs a high-performance key-value storage system based on persistent memory. However, the system has the problems of high CPU resource usage and increased CPU computing burden. SUMMARY

[0009] In view of the above problems, the purpose of the present application is to provide a FPGA accelerated parallel computing method for a Key-Value storage system, which offloads all the computationally intensive tasks of the compaction sub-operation (i.e. decoding, merging sorting and encoding) to FPGA, utilizes the parallelism of FPGA to improve the efficiency of garbage collection and reduce the CPU burden.

[0010] The FPGA accelerated parallel computing method for a Key-Value storage system provided by the embodiments of the present application comprises:

[0011] S1, the Key-Value storage system transmits the SSTable with repeated key ranges directly to the FPGA memory by triggering the compaction operation, and bypasses the CPU through the data transmission module;

[0012] S2, the FPGA is used to perform parallel decoding and sorting operations on the SSTable with repeated key ranges to obtain a set of key-value pairs to be sorted, and the FPGA is used for sorting and deduplication to obtain a set of ordered and non-repeated key-value pairs;

[0013] S3, using FPGA to perform parallel encoding operation on the ordered and non-repeated key-value pair set, forming a new SSTable file, and directly writing the new SSTable file to the SSD;

[0014] S4, performing parallel garbage collection operation to recover invalid key-value pairs in the log file.

[0015] Further, the S2 uses FPGA to perform parallel decoding and sorting operation on the SSTable with key range repetition, including the steps of:

[0016] S21, FPGA parallel decodes the footer, index block and data block of all SSTable, and stores them into the key-value pair set to be sorted;

[0017] S22, FPGA performs parallel merge sorting operation on the key-value pair set to be sorted, to obtain an ordered but possibly repeated key-value pair set;

[0018] S23, FPGA performs deduplication operation on the sorted key-value pair, to obtain an ordered and non-repeated key-value pair set.

[0019] Further, the S3 uses FPGA to perform parallel encoding operation on the ordered and non-repeated key-value pair set, including the steps of:

[0020] S31, dividing the ordered and non-repeated key-value pair set, and combining these key-value pairs into multiple SSTable to form an SSTable group;

[0021] S32, using FPGA to perform parallel encoding operation on the key-value pair set in the SSTable group, to form a new SSTable file;

[0022] S33, directly writing the new SSTable file from the FPGA memory to the SSD, bypassing the CPU;

[0023] Wherein, the FPGA parallel encoding process includes: parallel encoding data block, parallel encoding filter block, parallel encoding index block and parallel encoding other block.

[0024] Further, the S1 directly transmits the SSTable with key range repetition to the FPGA memory, bypassing the CPU through the data transmission module, and the steps are as follows:

[0025] CPU calls open() API to open the SSTable with key range repetition in the SSD;

[0026] The CPU instructs the SSD controller to transmit the SSTable file with key range repetition to the Direct Memory Access (DMA) engine;

[0027] The DMA engine directly transmits the SSTable to the FPGA memory without passing through the CPU.

[0028] Further, the new SSTable file is directly written from the FPGA memory to the SSD in the S3, bypassing the CPU, and the steps are:

[0029] The CPU calls the open() API to open the SSTable to be written in the SSD;

[0030] The CPU informs the FPGA driver to transmit the SSTable file to the DMA engine;

[0031] The DMA engine writes the SSTable file back to the SSD without passing through the CPU.

[0032] Further, the FPGA comprises:

[0033] An FPGA bitmap for marking invalid key-value pairs in the log file in parallel;

[0034] A parallel decoding, sorting and encoding module for performing parallel decoding, sorting and encoding operations on the selected SSTable to obtain a key-value pair set.

[0035] Further, the parallel garbage collection operation in the S4 recycles invalid key-value pairs in the log file, comprising the steps of:

[0036] The validity of each key-value pair in the log file corresponds to a bit of the FPGA bitmap, and the log files are continuously stored in the FPGA bitmap, and the P idx The address of each key-value pair in the FPGA bitmap is recorded, and the formula is:

[0037] P idx =P+(K-1)*N+idx

[0038] idx=pos / kv_size

[0039] Wherein, P is the first address of the FPGA bitmap, K is the log file number, N is the number of key-value pairs in the log file, and idx is the index position of the marked key-value pair validity bit;

[0040] When the FPGA is used to decode the repeated SSTable, the data bit corresponding to each key-value pair in the FPGA bitmap is marked in the log file, 0 representing invalid data and 1 representing valid data;

[0041] According to the validity of the FPGA bitmap data bit, the garbage collection operation performs parallel garbage collection on the invalid key-value pairs in the log file.

[0042] Further: the parallel coding and sorting module is provided with:

[0043] a decoding module for decoding the footer, index block and data block of SSTable in parallel to obtain a key-value pair set;

[0044] a sorting module for sorting the obtained key-value pair set to obtain an ordered key-value pair set that may have duplicates;

[0045] a deduplication module for deduplicating the duplicated key-value pairs in the sorted key-value pair set to obtain an ordered and non-duplicated key-value pair set;

[0046] an encoding module for encoding the ordered and non-duplicated key-value pair set to obtain SSTable.

[0047] Advantages of the present application:

[0048] 1. The FPGA-accelerated parallel computing method for key-value storage systems of the present application uses a key-value separation method to reduce the write amplification problem of key-value storage and provides comprehensive optimization for the computing load and data transmission delay in key-value storage.

[0049] 2. For CPU computing load, the present application method offloads all the compaction sub-operations (i.e. decoding, merging sorting and encoding) of the computationally intensive tasks to FPGA; by utilizing the parallelism of FPGA, an efficient FPGA-adapted continuous memory allocation and FPGA-optimized key-value block architecture is designed; a parallel merging sorting algorithm specially designed for key-value storage is proposed to reduce the high computing load of CPU in the compaction process, so that the CPU can handle other tasks such as I / O operations.

[0050] 3. For data transmission delay, the present application method designs a data transmission module to realize a direct data transmission mechanism from a storage device (such as SSD) to the memory of FPGA, bypassing the CPU memory; this mechanism minimizes data transmission delay and reduces the pressure on CPU, thereby improving data transmission bandwidth.

[0051] 4. The present application method comprehensively optimizes the computing load and data transmission delay of key-value storage, fully utilizes the parallelism of FPGA to improve the performance of key-value storage. In addition, for the new CPU burden brought by the key-value separation strategy, i.e. garbage collection, the present application uses FPGA to parallelly process the invalid key-value pair marking and garbage collection process of compaction to accelerate the invalid key-value pair marking and garbage collection operation, utilizes the parallelism of FPGA to improve the efficiency of garbage collection and reduce the CPU burden. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 This is a schematic diagram of the existing compaction operation process;

[0053] Figure 2 This is a schematic diagram illustrating the principle of FPGA parallel computing in this invention;

[0054] Figure 3 This is a schematic diagram of the compaction operation process of the method of the present invention;

[0055] Figure 4 This is a schematic diagram of the log file recycling process according to the method of the present invention;

[0056] Figure 5 This is a schematic diagram illustrating the decoding, sorting, deduplication, and encoding process of the method of the present invention. Detailed Implementation

[0057] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar symbols denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0058] To make it easier to understand, we will first introduce the standard operation process of compaction.

[0059] like Figure 1 As shown, in RocksDB's LSM-tree, the Put operation first writes the key-value pairs to the WriteAhead Log (WAL), then stores the key-value pairs in the memtable in memory. When the memtable is full, it is converted into an Immutable Memtable and added to the Flush queue. Subsequently, it is written to disk and converted into a Sorted String Table (SSTable).

[0060] The Get operation first searches for Mmetable and Immutable Memtable in memory. If it does not find them, it searches layer by layer in the LSM-tree in the SSD.

[0061] To maintain the high performance of LSM-tree, LSM-tree needs to trigger compaction operations periodically to merge duplicate data and clean up useless old data.

[0062] In RocksDB, the compaction operation first selects overlapping SSTables in the LSM-tree, choosing SSTables from adjacent levels of the LSM-tree. This process initializes a min-heap structure by creating an iterator pointing to the first key of each SSTable, and then performs a merge operation based on this. This process deletes old keys, retains the newest keys, and writes these new keys to new SSTables. Finally, after all new keys have been written, the old SSTables are deleted.

[0063] The compaction process in existing key-value storage systems increases CPU load, thus creating a system performance bottleneck.

[0064] To address the above problems, this invention provides an FPGA-accelerated parallel computing method for key-value storage systems. Figure 3 This is a flowchart illustrating the method of the present invention, which includes the following steps:

[0065] S1, the Key-Value storage system directly transfers SSTables with duplicate key ranges to the FPGA memory by triggering a compaction operation, bypassing the CPU through the data transmission module;

[0066] S2. Use FPGA to perform parallel decoding and sorting operations on SSTable with repeated key ranges to obtain a set of key-value pairs to be sorted, and use FPGA to sort and remove duplicates to obtain an ordered and non-repeating set of key-value pairs.

[0067] S3. Use FPGA to perform parallel encoding operations on the ordered and non-repeating key-value pair set to form a new SSTable file, and write the new SSTable file directly to SSD;

[0068] S4. Parallel garbage collection operations reclaim invalid key-value pairs from the log file.

[0069] like Figure 2 As shown, when a key-value pair arrives, the system performs key-value separation. First, the key and value information are stored in the log file. Second, the value pointer is encoded. The value pointer contains the value size, the file number of the key-value pair in the log file, and the position in the log file. Finally, the key and value pointer are written to the Memtable. When the Memtable is full, it is written to the SSTable in the LSM-tree on the disk.

[0070] The SSTable is first written to the L0 layer of the LSM-tree, and when the number of SSTables in the L0 layer reaches a threshold value triggering compaction, the compaction operation is triggered; the compaction operation first selects SSTables with repeated key ranges, and then uses FPGA to decode all input files.

[0071] The application sets a data transmission module between the FPGA and the SSD, which is a CPU-managed host component, and aims to realize CPU-free data bypass to improve data transmission efficiency. The module can reduce data transmission delay by realizing direct transmission of SSTables between the SSD and the FPGA. At the same time, the module directly transmits the SSTables with repeated key ranges from the SSD to the FPGA memory, and directly writes the newly generated SSTables from the FPGA memory back to the SSD. This direct data transmission mechanism can improve the performance of key-value storage by reducing transmission delay and reducing CPU load.

[0072] When reading the SSTable file, first, the CPU calls the open() API to open the file to be accessed. Second, the CPU instructs the FPGA driver to allocate an FPGA memory buffer for storing data in the SSTable. Next, the CPU instructs the SSD controller to transmit the SSTable to the Direct Memory Access (DMA) engine. Finally, the DMA engine directly transmits the SSTable to the FPGA memory, bypassing the CPU.

[0073] When writing a file, first, the CPU calls the open() API to open the SSTable to be written. Then, the CPU notifies the FPGA driver to transmit the processed data to the DMA engine. Finally, the DMA engine writes the data in the FPGA memory to the SSD, without involving the CPU. This mechanism minimizes data transmission delay and reduces the pressure on the CPU, thereby improving data transmission bandwidth.

[0074] By redesigning the data block, the SSTable structure is adapted to the parallelism of the FPGA. First, the original key without prefix shortening is stored, and each key is fixed to 24 bytes, of which the user key size is 16 bytes. This design ensures the uniformity of the size of the key-value pair in the data block, thereby realizing the consistency of the data block size. In addition, the structure of the index block is improved to ensure that the size of each data block handle is uniform, i.e., 10 bytes. The data block handle contains the encoded data block size and offset. Through these optimization designs, the FPGA can realize parallel encoding and decoding of multiple SSTables, greatly improving the encoding and decoding efficiency.

[0075] Meanwhile, to make the FPGA efficiently adapt to the SSTable structure, a parallel encoding and sorting module is arranged in the FPGA.

[0076] As shown in Figure 5 The parallel encoding and sorting module in the FPGA memory performs decoding footer, decoding index block and decoding data block operations in parallel to obtain all the key-value pairs of the SSTable and store them in a set. Next, the FPGA is used to perform parallel merge sorting on the key-value pairs to obtain an ordered key-value pair set that may have duplicates, and then the FPGA is used to perform a deduplication operation on the sorted key-value pairs to obtain an ordered and non-duplicate key-value pair set.

[0077] Specifically, the parallel encoding and sorting module divides the decoding into three sub-steps of parallel decoding footer, decoding index block and decoding data block.

[0078] The key-value pairs in the SSTable are stored in the data block, and the position of the data block is recorded in the index block. Each SSTable contains a footer, and the offset and size of the index block can be obtained by decoding the footer. When processing a group of SSTables, an equal number of FPGA threads are needed to decode the footer. The purpose of decoding the index block is to determine the position of the data block in the SSTable.

[0079] The index block is composed of multiple index items, each of which records the offset and size of the data block. When decoding the index block, the FPGA circuit is designed so that one group of logic blocks of the FPGA performs decoding of the SSTable, and another group of logic blocks performs decoding of the index block, achieving parallel decoding of multiple SSTables.

[0080] After decoding the index block, the position of the data block is obtained. Decoding the data block uses the same logic block allocation strategy as decoding the index block to decode all the data blocks of the SSTables in parallel, obtaining a key-value pair set to be sorted.

[0081] This design uses Sort, Unique and Delete functions suitable for FPGA to jointly complete the sorting and deduplication tasks of the key-value pair set to be sorted. The Sort function performs parallel sorting of the key-value pair set to be sorted in dictionary order. The Unique function moves the older key in the duplicate key-value pair to the end of the set. After the Unique function, the Delete function operation removes these duplicate items at the end of the set. These operations are all performed in parallel to improve the efficiency of the calculation. After performing these operations, an ordered and non-duplicate key-value pair set is obtained.

[0082] Before encoding, the codec and sorting module divides the ordered and non-repeated key-value pair set according to the SSTable size to divide a plurality of SSTables, referred to as SSTable groups, and a proper SSTable size can balance parallel computing resources and memory occupation, realize performance optimization, and too small size will reduce parallelism, and too large size will consume too much FPGA memory.

[0083] The SSTable groups are encoded, the encoding order is opposite to the decoding order, the FPGA is used to encode the SSTable groups in parallel, and the parallel encoding process is divided into parallel encoding of data blocks, encoding of filter blocks, encoding of index blocks, and finally encoding of other blocks (i.e. meta index blocks, attribute blocks and footer), to form a new SSTable file, and finally the files are directly written back to the SSD through the data transmission module.

[0084] The codec and sorting module encodes the data blocks first, and encodes a plurality of SSTables in parallel; then, the filter blocks are encoded, the filter blocks can improve the read performance of the system, when a key is queried, if the key does not exist in an SSTable, the SSTable is skipped, and the next SSTable is queried, effectively reducing disk I / O and improving read performance.

[0085] In the compaction process, invalid data in the LSM-tree is cleaned up, but invalid data in the log file is not cleaned up, so a garbage collection operation is needed; in order to reduce the frequency of garbage collection, the system sets a threshold for triggering garbage collection, and when the invalid data in the log file reaches the threshold, garbage collection is performed.

[0086] As shown in Figure 4 When the invalid data in a log file reaches a certain threshold, garbage collection is triggered, and the FPGA is used to perform garbage collection, in particular:

[0087] An FPGA bitmap is set in the FPGA, and the FPGA bitmap is used to mark invalid key-value pairs in parallel, in the sorting process, invalid key-value pairs are marked, whether the key is a new key or an old key is judged according to the sequence number of the key, and the old key is marked as an invalid key, and the invalid key is cleaned up to realize parallel invalid data marking of the FPGA.

[0088] The FPGA bitmap is a linear structure in the FPGA memory, each key-value pair in the log file occupies one bit in the FPGA bitmap, and all bits are stored continuously; the FPGA bitmap uses 0 to represent an invalid key-value pair and 1 to represent a valid key-value pair, and each bit only occupies 1 byte, so that a large number of key-value pairs are marked with small memory occupation.

[0089] In order to realize parallel marking of multiple log files in a bitmap by multi-threading in an FPGA parallel environment, the number of key-value pairs in a fixed log file is N.

[0090] By calculation, the position P' of a log file in a bitmap can be determined as follows:

[0091] P' = P + (K-1)*N

[0092] wherein P is the first address of the bitmap, K is the log file number (counting from 1), and N is the number of key-value pairs in the log file. Each log file occupies N bits in the bitmap, and the N bits are referred to as a block.

[0093] Next, in order to set the data bits in each block in the FPGA bitmap in parallel, the size of each key-value pair in the log file needs to be fixed as kv_size, so that the position of each key-value pair in the log file is linearly increased. The position pos of each key-value pair in the log file is:

[0094] pos = idx*kv_size

[0095] wherein idx is the index of the current key-value pair in the log file.

[0096] Therefore, the bit index of the key-value pair in the block is idx = pos / kv_size. Then the position P of a key-value pair in a log file in the FPGA bitmap is: idx

[0097] P idx = P + (K-1)*N + pos / kv_size

[0098] When performing parallel garbage collection on a certain log file by using an FPGA, in the FPGA bitmap, the design can parallelly determine whether the bit is valid by using the parallelism of the FPGA bitmap. When a bit in the block is 0, it indicates that the key-value pair corresponding to the bit is invalid, and then the FPGA thread exits without any processing. When the bit is 1, it indicates that the key-value pair corresponding to the bit is valid, and the key-value pair is written to an output buffer.

[0099] By the above method, all valid key-value pairs can be written to the output buffer in parallel. Finally, the key-value pairs in the output buffer are written back to the LSM-tree, and the parallel garbage collection operation is completed.

[0100] ​The application adopts a key-value separation strategy to reduce the write amplification problem existing in the KV store, optimizes the key-value storage system from the aspects of calculation load and data transmission delay, uses FPGA to accelerate the encoding and decoding and sorting process in the compaction operation process, and adopts a data pass-through method to optimize the data transmission path between the SSD and the FPGA.

[0101] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A parallel computing method for FPGA acceleration of a Key-Value storage system, characterized in that, The application relates to a method for parallel garbage collection of a key-value storage system, and belongs to the technical field of computer storage systems. S1, a Key-Value storage system directly transmits a key range repeated SSTable to FPGA memory by triggering a compaction operation, and bypasses a CPU through a data transmission module; S2, the FPGA is used to perform parallel decoding and sorting operations on the key range repeated SSTable, to obtain a to-be-sorted key-value pair set, and the FPGA is used to perform sorting and deduplication, to obtain an ordered and non-repeated key-value pair set; S3, the FPGA is used to perform parallel encoding operations on the ordered and non-repeated key-value pair set, to form a new SSTable file, and the new SSTable file is directly written into an SSD; S4, a parallel garbage collection operation is performed on invalid key-value pairs in a log file; In the S1, the key range repeated SSTable is directly transmitted to the FPGA memory, and the data transmission module bypasses the CPU, and the steps are as follows: A CPU calls an open() API to open the SSTable with key range repetition in an SSD; The CPU instructs an SSD controller to transmit the SSTable file with key range repetition to a direct memory access (Direct Memory Access, DMA) engine; The DMA engine directly transmits the SSTable to the FPGA memory without passing through the CPU; In the S4, the parallel garbage collection operation is performed on the invalid key-value pairs in the log file, and the steps are as follows: Using FPGA bitmap, the validity of key-value pair in each log file corresponds to a bit of FPGA bitmap, and the log files are stored continuously in FPGA bitmap, and the P idx The address of a key-value pair in each log file in FPGA bitmap is recorded, and the formula is represented as: P idx = P + (K - 1) * N + idx Idx = pos / kv_size wherein P idx is the position of a key-value pair in a log file in an FPGA bitmap, P is the first address of the FPGA bitmap, K is the log file number, N is the number of key-value pairs in the log file, idx is the index position of the key-value pair validity bit, pos is the position of each key-value pair in the log file, and kv_size is the size of each key-value pair in the log file. When the FPGA is used to perform decoding operations on the repeated SSTable, the data bits corresponding to each key-value pair in the FPGA bitmap are marked, 0 represents invalid data, and 1 represents valid data; According to the validity of the key-value pairs in the FPGA bitmap, the garbage collection operation is performed on the invalid key-value pairs in the log file in parallel.

2. The computational method of claim 1, wherein, In the S2, the FPGA is used to perform parallel decoding and sorting operations on the key range repeated SSTable, and the steps are as follows: S21, the FPGA parallel decodes the footer, index block and data block of all SSTables, and stores them into a to-be-sorted key-value pair set; S22, the FPGA performs parallel merge sorting operations on the to-be-sorted key-value pair set, to obtain an ordered key-value pair set which may have repetitions; S23, the FPGA performs deduplication operations on the sorted key-value pairs, to obtain an ordered and non-repeated key-value pair set.

3. The computational method of claim 1, wherein, In the S3, the FPGA is used to perform parallel encoding operations on the ordered and non-repeated key-value pair set, and the steps are as follows: S31, the ordered and non-repeated key-value pair set is divided, and the key-value pairs are combined into multiple SSTables to form an SSTable group; S32, the FPGA is used to perform parallel encoding on the key-value pair set in the SSTable group, to form a new SSTable file; S33, the new SSTable file is directly written from the FPGA memory to the SSD, bypassing the CPU; The FPGA parallel encoding process is as follows: parallel encoding of a data block, parallel encoding of a filter block, parallel encoding of an index block and parallel encoding of other blocks.

4. The computational method of claim 1, wherein: The new SSTable file is directly written from the FPGA memory to the SSD in S3, bypassing the CPU, and the steps are as follows: The CPU calls the open() API to open the SSTable to be written in the SSD; The CPU notifies the FPGA driver to transfer the SSTable file to the DMA engine; The DMA engine writes the SSTable file back to the SSD without passing through the CPU.

5. The computational method of claim 1, wherein: The FPGA comprises: An FPGA bitmap for marking invalid key-value pairs in the log file in parallel; A parallel decoding, sorting and encoding module for performing parallel decoding, sorting and encoding operations on the selected SSTable to obtain a key-value pair set.

6. The computational method of claim 5, wherein: The parallel decoding, sorting and encoding module comprises: A decoding module for decoding the footer, index block and data block of the SSTable in parallel to obtain a key-value pair set; A sorting module for sorting the obtained key-value pair set to obtain an ordered key-value pair set that may have duplicates; A deduplication module for deduplicating the duplicated key-value pairs in the sorted key-value pair set to obtain an ordered and non-duplicated key-value pair set; An encoding module for encoding the ordered and non-duplicated key-value pair set to obtain an SSTable.

Images