A key-value database storage method and system based on CSD

Abstract

The application relates to the technical field of data storage, and discloses a CSD-based key-value database storage method and system, which comprises the following steps: in response to a user inserting a key-value pair into a key-value database, storing the key-value pair into an in-memory table, and converting the in-memory table into an immutable in-memory table after a preset trigger condition is met; aligning and encoding the key-value pairs in the immutable in-memory table according to a uniform length to generate fixed-step key-value pairs; calling a computing storage device to automatically compress and flush the fixed-step key-value pairs into an SSTable file; receiving a trigger merge operation instruction, controlling the computing storage device to acquire the SSTable file and decompress the SSTable file; calling a graphics processing unit to perform merge processing on the decompressed file to generate a new SSTable file, and calling the computing storage device to automatically compress and flush the new file to a disk. The application significantly improves the read-write throughput and merge efficiency of the key-value database, and effectively alleviates the write blocking and write pausing problems.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, for example, to a CSD-based key-value database storage method and system. Background Art

[0002] Key-value databases based on LSM trees (Log-Structured Merge-trees) are widely used for high-performance data storage. They offer significant advantages in write-intensive and highly concurrent environments and are a core component of many big data applications. When writing data, key-value pairs (KV pairs) are first stored in a memtable (MemTable) and sorted by key size. When the memtable capacity reaches a threshold, it is converted to an immemtable and subsequently written to disk in batches using a sorted string table (SSTable), effectively reducing random disk writes.

[0003] In its operating mechanism, the LSM tree optimizes data writing efficiency and storage space utilization through layered and log-structured merge strategies. To maintain the hierarchical structure, the LSM tree needs to perform a large number of merge operations, merging and reorganizing data from different levels to maintain data consistency and reduce write amplification. However, the merge process is computationally intensive, involving a large amount of data movement and rewriting, consuming computing resources and disk I / O bandwidth. The system will experience short periods of write blockage, known as "write pauses," and the execution of merge operations may need to continue in the background. Especially in high-concurrency environments, merges are frequent and complex, affecting the overall throughput of the system. Therefore, there is an urgent need for a key-value database storage method to solve the above problems.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0005] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical elements or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0006] The embodiments of the present disclosure provide a CSD-based key-value database storage method and system to solve the technical problems of write obstruction and heavy computation in the operation mechanism of the LSM tree.

[0007] In some embodiments, a CSD-based key-value database storage method includes the following steps:

[0008] In response to a user inserting a key-value pair into the key-value database, the key-value pair is stored in a memory table, and after a preset trigger condition is met, the memory table is converted into an immutable memory table;

[0009] Align the key-value pairs in the immutable memory table with a uniform length and generate fixed-step key-value pairs.

[0010] Call the computing storage device to automatically compress the fixed-step key-value pairs and write them as SSTable files;

[0011] Receive the instruction to trigger the merge operation, control the computing storage device to obtain the SSTable file and decompress it;

[0012] Call the graphics processing unit to merge the decompressed SSTable files to generate new SSTable files;

[0013] Call the computing storage device to automatically compress the new SSTable file and flush it to disk.

[0014] In some embodiments, encoding the key-value pairs in the immutable memory table according to a uniform length alignment to generate fixed-step key-value pairs includes:

[0015] Compare the key in the key-value pair to be encoded with the maximum key length supported by the key-value database, and fill the insufficient part with zero;

[0016] Compare the value in the key-value pair to be encoded with the maximum length supported by the key-value database, and fill the insufficient part with zero.

[0017] In some embodiments, calling a computing storage device to automatically compress a key-value pair with a fixed step size includes:

[0018] The built-in hardware compression engine of the computational storage device is called to automatically compress consecutively filled zeros, and the compression process is transparent to the host system.

[0019] In some embodiments, calling a graphics processing unit to merge the decompressed SSTable files to generate a new SSTable file includes:

[0020] Multiple SSTable files are processed simultaneously through multi-threading. Each thread calculates the key-value pair address based on a fixed step size to separate the key and value.

[0021] According to the multi-threaded parallel sorting algorithm, the positions of the minimum keys and corresponding values ​​in multiple threads are output one by one to sort the keys;

[0022] Get the corresponding value according to the position of the corresponding value of each key, and combine each key-value pair according to the sorting result to generate a new SSTable file.

[0023] In some embodiments, the formula for calculating the address of a key-value pair based on a fixed step size is as follows:

[0024]

[0025] Where, the base address is the starting address of the data block, is the index number of the key-value pair, is the maximum key length supported by the key-value database, It is the maximum value length supported by the key-value database.

[0026] In some embodiments, after separating the key and the value, the method further comprises:

[0027] The separated keys are input into the sorter through the first FIFO buffer, and the separated values ​​are directly input into the encoder through the second FIFO buffer.

[0028] In some embodiments, according to a multi-threaded parallel sorting algorithm, outputting the positions of the minimum keys and corresponding values ​​in multiple threads one by one includes:

[0029] The control sequencer dynamically selects the global minimum key by comparing the head keys of multiple first FIFO buffers in parallel;

[0030] When the global minimum key is selected, the corresponding first FIFO buffer is controlled to output the minimum key, and the sorter is controlled to output the position of the value corresponding to the minimum key in the second FIFO buffer.

[0031] In some embodiments, when calculating the key-value pair address according to the fixed step size, the method further includes:

[0032] The starting address of each key-value pair is obtained and an index table is generated, wherein the index table is used to record the position of each key-value pair in the data block to locate the key-value pair.

[0033] In some embodiments, after calling the computing storage device to automatically compress and flush the new SSTable file to disk, the method further includes:

[0034] Organize and record the execution data of the merge operation, where the execution data includes the file information and compression ratio involved in the merge.

[0035] In some embodiments, a CSD-based key-value database storage system is used to execute any of the above-mentioned CSD-based key-value database storage methods.

[0036] The CSD-based key-value database storage method and system provided by the embodiments of the present disclosure can achieve the following technical effects:

[0037] When a user needs to store data in a key-value database, an instruction for representing the stored key-value pairs is sent to the system. In response to the user's instruction, the key-value pairs are first stored in a memory table. After the preset trigger condition is met, the memory table is converted into an immutable memory table. In this application, the key-value pairs in the immutable memory table are aligned and encoded according to a uniform length to generate key-value pairs with a fixed step size. By uniformly aligning the space occupied by each key-value pair, the dependencies between the data are eliminated, so that the data can be processed in parallel. At the same time, this application can use the computing storage device to compress the redundant data in the fixed-step key-value pairs and flush them to SSTable files. When a merge operation is triggered, the system receives the instruction to trigger the merge operation, obtains the SSTable file through the computing storage device, and decompresses it. Since the decompressed key-value pairs are all fixed-step lengths, they occupy the same space during storage and reading, so that each key-value pair can be processed independently and is no longer affected by the previous key-value pair. This eliminates the dependencies between the data and can be processed in parallel. Therefore, the parallel computing capabilities of the graphics processing unit can be fully utilized, and a large amount of data can be processed in a short time by the graphics processing unit, greatly accelerating the merge process. After the data is merged, a new SSTable file is generated. The system can call the computing storage device to automatically compress the new SSTable file and flush it to the disk. As mentioned above, this application significantly improves the read and write throughput and merge efficiency of the key-value database, effectively alleviating the problems of write blocking and write pause.

[0038] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] One or more embodiments are exemplarily described by corresponding drawings. These exemplary descriptions and drawings do not limit the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation. In addition,

[0040] Figure 1 This is a flowchart of a key-value database storage method based on CSD provided by an embodiment of the present disclosure;

[0041] Figure 2 This is a schematic diagram of a key-value pair uniform length alignment encoding provided by an embodiment of the present disclosure;

[0042] Figure 3 This is a flowchart of a key-value pair uniform length alignment encoding provided by an embodiment of the present disclosure;

[0043] Figure 4 This is a timing diagram of a key-value database storage method based on CSD provided by an embodiment of the present disclosure;

[0044] Figure 5This is a flowchart of a graphics processing unit merging and processing SSTable files provided by an embodiment of the present disclosure;

[0045] Figure 6 This is a design diagram of a graphics processing unit merging acceleration core provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure is described in detail below in conjunction with the accompanying drawings. The accompanying drawings are for reference only and are not used to limit the embodiments of the present disclosure. In the following technical description, for the sake of convenience of explanation, a full understanding of the disclosed embodiments is provided through multiple details. However, one or more embodiments can still be implemented without these details. In other cases, to simplify the drawings, well-known structures and devices can be simplified for display.

[0047] The terms "first," "second," and the like in the embodiments of the present disclosure are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate to facilitate the description of the embodiments of the present disclosure herein. Furthermore, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions.

[0048] Unless otherwise stated, the term "plurality" means two or more.

[0049] In the embodiment of the present disclosure, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0050] The term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0051] The term "correspondence" may refer to an association relationship or a binding relationship. The correspondence between A and B means that there is an association relationship or a binding relationship between A and B.

[0052] The LSM tree key-value store, due to its efficient write performance and space optimization, has become a core storage solution for write-intensive, high-concurrency scenarios such as big data. Sequenced string tables (SSTables) on disk are organized into a multi-level hierarchy. Lower levels typically adopt a tier strategy, allowing overlapping key ranges within SSTables; higher levels adopt a level strategy, requiring that SSTables within a tier be stored in key order with non-overlapping ranges. Through this hierarchical compaction strategy, the LSM tree optimizes storage space utilization while ensuring query efficiency. However, LSM tree merge operations require frequent reorganization of hierarchical data to maintain consistency. These computationally intensive merges involve significant data movement and rewriting, consuming computing resources and disk I / O bandwidth. This can lead to short periods of write blockage, known as "write pauses," and the system may need to continue executing merge operations in the background. Especially in high-concurrency environments, frequent and complex merges can affect overall system throughput.

[0053] In order to solve the above problems, the present disclosure provides a key-value database storage method based on CSD. The key-value database storage method provided by the present disclosure is described in detail below with reference to the accompanying drawings.

[0054] Figure 1 This is a flowchart of a key-value database storage method based on CSD provided by the embodiment of the present disclosure, combined with Figure 1 As shown, the key-value database storage method includes the following steps:

[0055] S101: In response to a user inserting a key-value pair into a key-value database, the key-value pair is stored in a memory table, and after a preset trigger condition is met, the memory table is converted into an immutable memory table.

[0056] In some embodiments, when a user initiates an insert operation into a key-value store based on an LSM tree, the system first writes the key-value pair to a mutable in-memory data structure, known as a MemTable. MemTables are typically implemented using a SkipList or a balanced tree (such as a B+ tree) to ensure key order. Once a pre-set trigger condition is met, the MemTable is converted to an immutable MemTable.

[0057] In some embodiments, the preset trigger conditions for triggering the memory table conversion include: the amount of data in the memory table reaches a preset upper limit, such as 256MB; a regular refresh mechanism, such as forced conversion every 5 minutes; some system events occur, such as the write buffer is too high or the background merge task is ready. Once any of the above conditions is met, the system will freeze the current memory table and mark it as a read-only immutable memory table (Immutable MemTable), and immediately enable a new empty memory table to receive subsequent writes to ensure that the write is not blocked. After that, the background thread will sort the data in the immutable memory table by key and write it to the disk in batches to generate a persistent SSTable file.

[0058] S102: Align the key-value pairs in the immutable memory table according to a uniform length to generate key-value pairs with a fixed step length.

[0059] Figure 2 This is a schematic diagram of a key-value pair uniform length alignment encoding provided by an embodiment of the present disclosure. Figure 2 As shown in the left figure, in the traditional LSM tree, due to the size difference between key-value pairs, there is a dependency between the data in the SSTable file. When reading data, when reading an SSTable file, the previous key-value pair must be read before the starting position of the next key-value pair can be known. In other words, after reading Key1Value1, Key2Value2 can be read. The sequential dependency means that each read operation must wait for the previous operation to complete before it can proceed. This makes it impossible to fully utilize the advantages of concurrent devices and limits the introduction of high-concurrency devices.

[0060] In some embodiments, the system encodes the key-value pairs in the immutable memory table in a uniform length alignment to generate key-value pairs with a fixed step length. Figure 2 As shown in the figure on the right, all keys are of the same length, and all values ​​are of the same length. Therefore, the length of each key-value pair in the SSTable file is fixed to the same size. Usually, the fixed step size here is set to the sum of the maximum key length and the maximum value length supported by the key-value database.

[0061] In some embodiments, the key in the key-value pair to be encoded is compared with the maximum key length supported by the key-value database, and the insufficient part is filled with zeros; the value in the key-value pair to be encoded is compared with the maximum value length supported by the key-value database, and the insufficient part is filled with zeros.

[0062] Figure 3 This is a flowchart of a key-value pair uniform length alignment encoding provided by an embodiment of the present disclosure. Figure 3, after receiving the key-value pair, the lengths of the key and the value are judged respectively. If the length of the current Key is less than the maximum Key length supported by the key-value database, the current Key length is filled with zeros. If the length of the current Value is less than the maximum Value length supported by the key-value database, the current Value length is filled with zeros. Of course, if the length of the current key-value pair is the above fixed step length, it is directly combined into a key-value pair.

[0063] For example, if the maximum Key length supported by the key-value database is 16 bytes and the maximum Value length is 128 bytes, then each key-value pair occupies:

[0064]

[0065] For an actually stored key-value pair (Key = 8 bytes, Value = 86 bytes pair), it is necessary to fill 0 for (16 - 8) + (128 - 86) = 50 bytes:

[0066]

[0067] For another example, assume the maximum Key = 16 bytes (e.g., "Zhang San" occupies 4 bytes, and 12 zeros are filled to 16 bytes), the maximum Value = 128 bytes (e.g., "math book" occupies 6 bytes, and 122 zeros are filled to 128 bytes), and the total length is 144 bytes.

[0068] In this way, each key-value pair is completely consistent in the storage structure, enabling the data reading and writing to be sliced according to the fixed step length, and subsequent introduction of high-concurrency devices can process these data in parallel.

[0069] As above, in this application, regardless of the actual size of the data itself, by aligning and encoding all key-value pairs according to the unified length, making them occupy the same space during storage and reading, each key-value pair can be processed independently, no longer affected by the previous key-value pair, eliminating the dependency relationship between data, so that the data can be processed in parallel, greatly improving the concurrency and throughput of data reading and writing.

[0070] S103: Call the computing storage device to automatically compress and flush the key-value pairs with a fixed step length into an SSTable file.

[0071] Computational Storage Device (CSD) is a groundbreaking storage architecture that deeply integrates computing units with storage media to address storage bottlenecks in big data and high-performance computing scenarios. CSDs allow the host to deploy dedicated accelerator cores and directly control data transmission and interaction between the accelerator and storage media. Because the computing accelerator and storage media are integrated into the same device, data transmission between them enjoys the advantages of high bandwidth, low latency, and low power consumption, significantly reducing data transfer overhead and host resource utilization.

[0072] In some embodiments, taking the ScaleFlux CSD 3000 series solid-state drives as an example, the computational storage device has the important feature of transparent compression. When data is written to the computational storage device, the hardware-level compression engine eliminates invalid and redundant data in real time, converts the original data into a compact encoding format, completely avoids host resource consumption, and does not require upper-layer application adaptation, thereby reducing physical storage space requirements from the root. In response to a data read request, the hardware decoder is automatically triggered to restore the original data and provide it to the application in its original form. This seamless compression and decompression mechanism makes the data reading process unaffected and minimizes latency. At the same time, the data compression and decompression process is completely transparent to the application layer, and users and applications do not need to worry about whether the data is compressed or how to decompress it.

[0073] In some embodiments, when data is written, key-value pairs are aligned and encoded to a uniform length, resulting in a large amount of redundant data (padding with zeros). The system can then invoke the computational storage device's built-in hardware compression engine to automatically compress the consecutive zero padding. Since these redundant zeros are stored consecutively, the compressed space occupied by these zeros is minimal, minimizing the storage space occupied by invalid data. Furthermore, even small data sizes are physically aligned to a predetermined fixed length. This process is automatically performed by the CSD device at the hardware level and is transparent to the host system, eliminating the need for the application layer and database to be aware of the details of data padding and compression.

[0074] In some embodiments, when the length of the continuous padding zeros exceeds a preset threshold, such as exceeding 4kb, the hardware compression engine of the computing storage device may be triggered to perform a batch compression operation.

[0075] S104: Receive a trigger merge operation instruction, control the computing storage device to obtain the SSTable file and decompress it.

[0076] In some embodiments, upon receiving a triggering merge operation (Compaction) instruction, the computing storage device obtains the SSTable file and automatically decompresses the key-value pairs filled with 0 to the original size, so that the key-value database can return complete key-value pair data in the expected format.

[0077] In this application, by leveraging the computational compression characteristics of computational storage devices, key-value pair read and write operations can be processed with high concurrency within the graphics processing unit (GPU) without blocking other concurrent operations. Integrating the data writing and compression processes within the storage device avoids the burden of relying on the CPU for compression and decompression in traditional storage.

[0078] S105: Call the graphics processing unit to merge the decompressed SSTable files to generate new SSTable files.

[0079] Figure 4 This is a sequence diagram of a key-value database storage method based on CSD provided by an embodiment of the present disclosure. Figure 4 , the background thread triggers Compaction and starts the data merging related process. The merge thread selects the files that need to participate in the Compaction so that the computing storage device can obtain the relevant files. As in step S104, when receiving the instruction to trigger the merge operation (Compaction), the computing storage device will obtain the SSTable file, decompress the obtained SSTable file, and complete the length of the key-value pairs in the SSTable file. After obtaining the decompressed file, the merge thread transfers the file to the graphics processing unit for Compaction.

[0080] In some embodiments, the system calls the graphics processing unit to perform compaction in three steps, namely decoding (writing), sorting, and encoding (reading). Figure 5 This is a flowchart of the graphics processing unit merging and processing SSTable files provided by the embodiment of the present disclosure, combined with Figure 5 , the following is a detailed introduction to the process of merging and processing SSTable files.

[0081] S501: Multiple SSTable files are processed simultaneously through multi-threading. Each thread calculates the key-value pair address based on a fixed step size to separate the key and value.

[0082] Figure 6 This is a design diagram of a graphics processing unit merging an acceleration core provided by an embodiment of the present disclosure. Figure 6The computing storage device (CSD Device) decompresses the obtained SSTable (SST) file, and the decompressed SSTable file enters the graphics processing unit decoder (GPU Decoder). The graphics processing unit decoder first splits the SSTable file into data blocks (Data Block), and then parses the data blocks into key-value pairs (KVpair) to prepare for subsequent sorting, and uses the parallel computing capabilities of the GPU to accelerate the decoding process.

[0083] In some embodiments, since the storage length of all key-value pairs is consistent, the graphics processing unit can directly calculate the starting address of each key-value pair after parsing the location of each data block. The calculation formula for calculating the key-value pair address based on a fixed step size is as follows:

[0084]

[0085] Where, the base address is the starting address of the data block, is the index number of the key-value pair, is the maximum key length supported by the key-value database, This is the maximum length supported by the key-value database. For example, if the base address is 1000, the maximum key length is 16 bytes, and the maximum value length is 128 bytes. The address of the third KV pair is 1000 + 3 × (16 + 128) = 1432, which can be read directly without traversal.

[0086] After parsing the key-value pairs, the GPU decoder separates the key and value, inputs the separated key into the sorter through the first FIFO buffer (Key FIFO), and directly inputs the separated value into the encoder through the second FIFO buffer (Value FIFO).

[0087] In some embodiments, when calculating the key-value pair address based on a fixed step size, the storage starting address of each key-value pair can also be obtained and an index table can be generated, wherein the index table is used to record the position of each key-value pair in the data block for use by the sorter to quickly locate the key-value pair.

[0088] S502: According to the multi-thread parallel sorting algorithm, the positions of the minimum keys and corresponding values ​​in the multi-threads are outputted one by one to sort the keys.

[0089] In some embodiments, a graphics processing unit sorter (GPU Sorter) receives key-value pairs output by a graphics processing unit decoder, sorts all keys, and records positions of corresponding values.

[0090] according to Figure 6As shown, the GPU sequencer can read key data from multiple encoders in parallel and use multiple first FIFO buffers to store the keys. The sequencer dynamically selects the global minimum key by comparing the head keys of the multiple first FIFO buffers in parallel. When the global minimum key is selected, the corresponding first FIFO buffer is controlled to output the minimum key, and the sequencer is controlled to output the position of the value corresponding to the minimum key in the second FIFO buffer.

[0091] S503: Obtain the corresponding value according to the position of the corresponding value of each key, and combine each key-value pair according to the sorting result to generate a new SSTable file.

[0092] In some embodiments, the graphics processing unit encoder (GPU Encoder) receives keys arranged in sequence from the sorter, and reads corresponding values ​​from the Value FIFO according to the location information of the corresponding values ​​as instructed by the sorter, builds DataBlock (data block) and IndexBlock (index block), and finally encodes to generate a new SSTable file.

[0093] S106: Call the computing storage device to automatically compress the new SSTable file and write it to the disk.

[0094] In some embodiments, the merge operation is performed using a differentiated strategy based on the SSTable level. A Tier merge strategy is used for low-level SSTable files, allowing overlap in the merged key value ranges. A Level merge strategy is used for high-level SSTables to ensure that the key value ranges within the layer are strictly ordered and non-overlapping.

[0095] In some embodiments, the computational storage device automatically compresses the encoded new SSTable file and flushes it to disk. During the write process, because the Key FIFO and Value FIFO are read-only during the merge process, the trailing zero padding does not require the GPU to intervene, maximizing the concurrent processing capabilities of the GPU.

[0096] In some embodiments, combined Figure 4 As shown, after calling the computing storage device to automatically compress and flush the new SSTable file to the disk, the method further includes: collating and recording the execution data of the merge operation for system monitoring and optimization. The execution data includes file information and compression ratio involved in the merge.

[0097] The CSD-based key-value database storage method in this application utilizes the transparent compression characteristics of computational storage devices (CSDs) to design a data encoding method that is friendly to high-concurrency device reads and writes, separates the dependencies between data, and introduces high-concurrency devices on this basis to write a concurrent processing accelerator kernel for merge operation decoding, merging, and encoding tasks. This greatly improves the read and write throughput, reduces the performance bottleneck faced by LSM trees in high-concurrency environments, and effectively alleviates the problems of write pauses and write blocking, enabling large-scale distributed key-value databases to maintain efficient read and write performance under high load conditions, and enabling the LSM tree architecture to better adapt to the needs of modern high-concurrency applications.

[0098] Based on the same inventive concept as the above-mentioned CSD-based key-value database storage method, the present application also discloses a CSD-based key-value database storage system in some embodiments. The system is used to execute the CSD-based key-value database storage method disclosed in any of the above-mentioned embodiments, and its process is not repeated here.

[0099] The technical solutions of the embodiments of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. The aforementioned storage medium may be a non-transitory storage medium, including: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, and other media that can store program code, or a transient storage medium.

[0100] The above description and the accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent only possible variations. Unless expressly required, individual components and functions are optional, and the order of operations may vary. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. Moreover, the terms used in this application are only used to describe the embodiments and are not used to limit the scope of protection. As used in the description herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to also include the plural forms. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of one or more associated listings. In addition, when used in this application, the term "comprise" and its variations "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, an element defined by the sentence "comprising a..." does not exclude the presence of other identical elements in the process, method or device that includes the element. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments can be referenced to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can be referred to the description of the method part.

[0101] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. The technicians will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0102] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices and equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units may be merely a logical functional division. In actual implementation, other division methods may be used, such as combining or integrating multiple units or components into another system, or omitting or disabling some features. In addition, the coupling or direct coupling or communication connection shown or discussed between each other may be through some interface, or the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to implement the present embodiments according to actual needs. In addition, the functional units in the embodiments of the present disclosure may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit.

Claims

1. A key-value database storage method based on CSD, characterized in that: The following steps are involved: In response to a user inserting a key-value pair into the key-value database, storing the key-value pair in a memory table, and converting the memory table into an immutable memory table after a preset trigger condition is met; Aligning and encoding the key-value pairs in the immutable memory table according to a uniform length to generate fixed-step key-value pairs, including: comparing the key in the key-value pair to be encoded with the maximum key length supported by the key-value database, and filling the insufficient part with zeros; comparing the value in the key-value pair to be encoded with the maximum length supported by the key-value database, and filling the insufficient part with zeros; Call the computing storage device to automatically compress the fixed-step key-value pairs and write them as SSTable files; Receive a trigger merge operation instruction, control the computing storage device to obtain the SSTable file and decompress it; Calling the graphics processing unit to merge the decompressed SSTable files to generate new SSTable files, including: processing multiple SSTable files simultaneously through multi-threading, each thread calculates the key-value pair address according to a fixed step size, and separates the key and value, wherein the calculation formula for calculating the key-value pair address according to the fixed step size is as follows: , where the base address is the starting address of the data block, is the index number of the key-value pair, is the maximum key length supported by the key-value database, It is the maximum length supported by the key-value database; according to the multi-threaded parallel sorting algorithm, the position of the minimum key and the corresponding value in multiple threads is output one by one to sort the keys; the corresponding value is obtained according to the position of the corresponding value of each key, and each key-value pair is combined according to the sorting result to generate a new SSTable file; Call the computing storage device to automatically compress the new SSTable file and write it to the disk.

2. The key-value database storage method based on CSD according to claim 1, characterized in that: The calling of the computing storage device to automatically compress the key-value pairs with a fixed step size includes: The built-in hardware compression engine of the computing storage device is called to automatically compress the continuously filled zeros, and the compression process is transparent to the host system.

3. The key-value database storage method based on CSD according to claim 1, characterized in that: After separating the keys and values, the method further includes: The separated keys are input into the sorter through the first FIFO buffer, and the separated values ​​are directly input into the encoder through the second FIFO buffer.

4. The key-value database storage method based on CSD according to claim 3, characterized in that: The method of outputting the positions of the minimum keys and corresponding values ​​in multiple threads one by one according to the multi-threaded parallel sorting algorithm includes: Controlling the sorter to dynamically select a global minimum key by comparing the head keys of multiple first FIFO buffers in parallel; When the global minimum key is selected, the corresponding first FIFO buffer is controlled to output the minimum key, and the sorter is controlled to output the position of the value corresponding to the minimum key in the second FIFO buffer.

5. The key-value database storage method based on CSD according to claim 1, characterized in that: When calculating the key-value pair address according to the fixed step size, the method further includes: The starting address of each key-value pair is obtained and an index table is generated, wherein the index table is used to record the position of each key-value pair in the data block to locate the key-value pair.

6. The key-value database storage method based on CSD according to claim 1, characterized in that: After calling the computing storage device to automatically compress and flush the new SSTable file to the disk, the method further includes: The execution data of the merge operation is sorted and recorded, wherein the execution data includes information of files involved in the merge and compression ratio.

7. A key-value database storage system based on CSD, characterized in that: The system is used to execute the CSD-based key-value database storage method described in any one of claims 1 to 6.

Images