Key value database near-storage computing system based on computing storage device

Abstract

The invention relates to the technical field of key value databases, and discloses a key value database near-storage computing system based on computing storage equipment, the system comprises a host, a memory and a computing memory, a decoder in the computing memory is used for reading files needing to be merged from the memory, analyzing all key value pairs from the files, and storing the key value pairs in the computing memory; placing the key value pairs in a buffer area connected with a sequencer; the sequencer is used for comparing the size relationship of the key value pairs, taking out the key value pair from the corresponding decoder buffer area according to the comparison result of each time, selecting the key value pair with the minimum key value according to the size of the key in the key value pair, and transmitting the corresponding key value pair to the buffer area connected with the encoder; and the encoder is used for extracting the key value pair from the buffer area connected with the sequencer, generating an SS-Table file according to a file encoding rule and storing the SS-Table file into the memory. Deep fusion of calculation and storage can be realized, calculation resources in a host are fully utilized, and the overall system performance is improved.

Description

Technical Field

[0001] This application relates to the technical field of key-value databases. For example, it relates to a key-value database near-storage computing system based on a compute storage device. Background Art

[0002] Databases play a crucial role in modern computer systems and are widely used, from enterprise data management to big data analysis. Key-value databases based on the Log-Structured Merge-tree (LSM-Tree) are widely adopted in many application scenarios due to their low write amplification and high write throughput. Representatives of such databases include LevelDB and RocksDB. The LSM-Tree effectively utilizes the sequential write characteristics of modern Solid State Drives (SSDs) by converting random write operations into sequential writes, thus significantly improving write performance. In addition, through frequent merge operations, the LSM-Tree not only ensures data consistency but also effectively reduces the write amplification problem. To improve read operation performance, the LSM-Tree maintains some auxiliary data structures in memory, reducing the number of Input / Output (I / O) accesses to the disk. It is these advantages that make the LSM-Tree the preferred choice for high-performance database design.

[0003] In the operation mechanism of the LSM-Tree, to maintain its hierarchical structure, a large number of merge operations need to be triggered. These operations merge and reorganize the data in different levels to maintain data consistency and reduce write amplification. However, merge operations involve a large number of I / O operations and data merge calculations, which consume a large amount of system resources. When the system resources are heavily occupied by background merge operations, the foreground write and read operations may be affected, resulting in frequent pauses of foreground write operations and a decrease in overall throughput.

[0004] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of this application. Summary of the Invention

[0005] To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review nor is it intended to identify key / important elements or delineate the protection scope of these embodiments, but rather serves as a preface to the subsequent detailed description.

[0006] Embodiments of the present disclosure provide a key-value database near-storage computing system based on a compute storage device, which solves the problem of insufficient overall system performance.

[0007] The system includes: a host CPU, a memory, and a computing memory. Among them, a memory and a computing accelerator are deployed in the computing memory, and the computing accelerator includes a decoder, a sorter, and an encoder; The decoder is used to read the SS-Table files that need to be merged from the memory, parse all key-value pairs from the files, and place the key-value pairs in the FIFO buffer connected to the sorter; The sorter is used to compare the size relationships of the key-value pairs, take out the key-value pairs from the corresponding decoder FIFO buffer according to the result of each comparison, select the key-value pair with the smallest key according to the size of the key in the key-value pair, and transmit the corresponding key-value pair to the FIFO buffer connected to the encoder; The encoder is used to take out the key-value pairs from the FIFO buffer connected to the sorter and generate SS-Table files according to the file encoding rules and store them in the memory.

[0008] In some embodiments, reading the SS-Table files that need to be merged from the memory, parsing all key-value pairs from the files, and placing the key-value pairs in the FIFO buffer connected to the sorter includes: Determine whether the reading of the current data block is completed. If it is completed, read the next data block from the memory into the data block buffer stored on the BRAM; Read the current data from the buffer and read the key-value pair metadata according to the current data; Read the key-value pair information according to the key-value pair metadata; Transmit the read key-value pairs to the FIFO buffer; Update the read index according to the current key-value pair metadata.

[0009] In some embodiments, comparing the size relationships of the key-value pairs, taking out the key-value pairs from the corresponding decoder FIFO buffer according to the result of each comparison, selecting the key-value pair with the smallest key according to the size of the key in the key-value pair, and transmitting the corresponding key-value pair to the FIFO buffer connected to the encoder includes: Read a key-value pair from the FIFO buffer connected to the decoder; Separate the read key-value pair to obtain the user key-value and timestamp in the key-value pair; Compare the sizes of all the obtained user key-values; Transmit the key-value pair with the smallest user key-value to the FIFO buffer connected to the encoder; Supplement key-value pairs from the decoder corresponding to the key-value pair with the smallest user key-value.

[0010] In some embodiments, key-value pairs are fetched from the FIFO buffer connected to the sorter, and SS-Table files are generated according to the file encoding rules and stored in the memory, including: Read key-value pairs from the FIFO buffer connected to the sorter; Determine the validity of the current key-value pair; Determine the encoding of the key-value in the current key-value pair according to the encoding status and the information of the previous key-value pair; Store the encoded key encoding into the buffer in the memory; Store the encoded value encoding into the buffer in the memory; Update the encoder status according to the current key-value pair.

[0011] In some embodiments, the system is also used for: Statistically analyze the metadata of the SS-Table files participating in the merge; Map the SS-Table file space in the dynamic memory part of the computing memory; Use P2P for transmission to transfer the SS-Table file from the computing memory storage module to the dynamic memory inside the computing memory; Pre-allocate a buffer for the merged data in the dynamic memory of the computing memory; Activate the computing units in the computing memory to perform merge calculations; The host CPU receives the return parameters returned by the computing units of the computing memory; The host CPU generates partial metadata of the target SS-Table file according to the parameters returned by the computing units; Use P2P for transmission to write the generated partial data of the SS-Table file from the dynamic memory of the computing memory back to the storage module of the computing memory.

[0012] In some embodiments, the host CPU further includes an intelligent merge operation trigger for dynamically triggering merge calculations according to the real-time load status of the system.

[0013] In some embodiments, dynamically triggering merge calculations according to the real-time load status of the system includes: Detect the real-time load status of the host CPU and the computing memory in the current system; According to the load status of the host CPU and the load status of the computing memory, determine the device type with a heavier workload and the priority level corresponding to the workload difference among the two types of devices, namely the host CPU and the computing memory, in the current system; According to the determined device type and priority level, trigger the merge trigger policy corresponding to the device and priority level; The actual computing device for predictive analysis calculation dynamically adjusts the SS-Table files included in a single merge calculation; Generate corresponding merge calculations for the key-value database.

[0014] In some embodiments, the system further includes a computing storage device intelligent dynamic scheduler for coordinating merge operations, task allocation, and resource allocation between the host CPU and the computing memory.

[0015] In some embodiments, the above-mentioned coordination of merge operations, task allocation, and resource allocation between the host CPU and the computing memory includes: After receiving the merge calculation feedback information triggered by the key-value database, determine the computing overhead of the current calculation on the host CPU and the computing memory based on the merge level of the generated calculation, the number of SS-Tables included, and the file size characteristics; Compare the computing time of the host CPU and the computing memory and the impact on the overall system resources to determine the computing device; Add the corresponding merge calculation to the calculation queue of the corresponding computing device; Regularly detect the calculation queues of all devices and dynamically adjust the allocated computing devices and computing priorities according to the waiting time in the calculation queues.

[0016] In some embodiments, the system further includes a key-value database parameter dynamic tuner, which is used to regularly detect the workload of the entire system; Analyze the detected workload and analyze the optimal strategy including the memory buffer and background merge thread parameters in the current database according to the tuning strategy; Adjust the parameters in the key-value database according to the analysis results.

[0017] The key-value database near-storage computing system based on a computing storage device provided by the embodiments of the present disclosure can achieve the following technical effects: The present disclosure significantly reduces the data transfer overhead between the disk and the host memory, improves the throughput rate of the merge calculation, and reduces the operation time delay. Moreover, the kernel is optimized for the merge operation, improving the overall performance. The scheduling strategy for the host-side and the computing memory to cooperate to accelerate the merge operation ensures the optimal utilization of computing and storage resources, further improving the system performance.

[0018] The above general description and the following description are only exemplary and explanatory, and are not used to limit this application. Brief Description of the Drawings

[0019] One or more embodiments are exemplarily illustrated by corresponding drawings. These exemplary illustrations and the drawings do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are shown as similar elements. The drawings do not constitute a scale limitation, and wherein: Figure 1 is a schematic structural diagram of a near-storage computing system of a key-value database based on a compute-storage device provided by an embodiment of the present disclosure; Figure 2 is a schematic overall diagram of a computing framework of a key-value database based on a compute-storage device provided by an embodiment of the present disclosure; Figure 3 is a schematic overall architecture diagram of a merged computing hardware accelerator kernel provided by an embodiment of the present disclosure; Figure 4 is a schematic structural diagram of a near-storage computing device of a key-value database based on a compute-storage device provided by an embodiment of the present disclosure. Detailed implementation manners

[0020] In order to be able to understand the features and technical content of the embodiments of the present disclosure in more detail, the implementation of the embodiments of the present disclosure will be described in detail below with reference to the drawings. The attached drawings are for reference and illustration only and are not used to limit the embodiments of the present disclosure. In the following technical descriptions, for the sake of explanation, numerous details are provided to give a thorough understanding of the disclosed embodiments. However, one or more embodiments may still be implemented without these details. In other cases, well-known structures and devices may be shown in a simplified manner.

[0021] The terms "first", "second", etc. in the embodiments of the present disclosure are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so as to implement the embodiments of the present disclosure described herein. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0022] Unless otherwise specified, the term "plurality" means two or more.

[0023] In the embodiments of the present disclosure, the character " / " indicates that the objects before and after are in an "or" relationship. For example, A / B means: A or B.

[0024] The term "and / or" is a description of the association relationship of an object and indicates that three relationships can exist. For example, A and / or B means: A or B, or, A and B these three relationships.

[0025] The term "corresponding" may refer to an association relationship or a binding relationship. A corresponding to B means that there is an association relationship or a binding relationship between A and B.

[0026] To solve the above problems, the present disclosure provides a near-storage computing system for a key-value database based on a compute storage device.

[0027] The following describes the near-storage computing system for a key-value database based on a compute storage device provided by the embodiments of the present disclosure with reference to the accompanying drawings.

[0028] Figure 1 It is a schematic structural diagram of a near-storage computing system for a key-value database based on a compute storage device provided by an embodiment of the present disclosure.

[0029] As Figure 1 shown, the near-storage computing system for a key-value database based on a compute storage device may include a host central processing unit (CPU) 101, a memory 102, and a compute memory 103. Among them, a memory 1031 and a compute accelerator 1032 are deployed in the compute memory 103, and the compute accelerator 1032 includes a decoder 10321, a sorter 10322, and an encoder 10323; The decoder 10321 is configured to read the SS-Table file that needs to be merged from the memory, parse all key-value pairs from the file, and place the key-value pairs in the FIFO buffer connected to the sorter; The sorter 10322 is configured to compare the magnitude relationship of the key-value pairs, and according to the result of each comparison, take out the key-value pairs from the corresponding decoder first-in-first-out (FIFO) buffer, select the key-value pair with the smallest key according to the magnitude of the keys in the key-value pairs, and transmit the corresponding key-value pair to the FIFO buffer connected to the encoder; The encoder 10323 is configured to take out the key-value pairs from the FIFO buffer connected to the sorter, and generate a sorted string table (SS-Table) file according to the file encoding rule and store it in the memory.

[0030] In some embodiments, reading the SS-Table file that needs to be merged from the memory, parsing all key-value pairs from the file, and placing the key-value pairs in the FIFO buffer connected to the sorter includes: Determining whether the reading of the current data block is completed. If it is completed, read the next data block from the memory to the data block buffer area stored on the block random access memory (BRAM); Reading the current data from the buffer, and reading the key-value pair metadata according to the current data; Reading the key-value pair information according to the key-value pair metadata; Transfer the read key-value pairs to the FIFO buffer; Update the read subscript according to the current key-value pair metadata.

[0031] In some embodiments, compare the size relationship of the key-value pairs, and take out the key-value pairs from the corresponding decoder FIFO buffer according to the result of each comparison, select the key-value pair with the smallest key according to the size of the key in the key-value pair, and transfer the corresponding key-value pair to the FIFO buffer connected to the encoder, including: Read a key-value pair from the FIFO buffer connected to the decoder; Separate the read key-value pair to obtain the user key value and timestamp in the key-value pair; Compare the sizes of all obtained user key values; Transfer the key-value pair with the smallest user key value to the FIFO buffer connected to the encoder; Supplement the key-value pair corresponding to the key-value pair with the smallest user key value from the decoder.

[0032] In some embodiments, take out the key-value pairs from the FIFO buffer connected to the sorter and generate SS-Table files according to the file encoding rules and store them in the memory, including: Read the key-value pairs from the FIFO buffer connected to the sorter; Determine the validity of the current key-value pair; Determine the encoding of the key value in the current key-value pair according to the encoding status and the information of the previous key-value pair; Store the encoded key encoding in the buffer in the memory; Store the encoded value encoding in the buffer in the memory; Update the encoder status according to the current key-value pair.

[0033] In some embodiments, Figure 1 The system in also used for: Statistical metadata of the SS-Table files participating in the merge; Map the SS-Table file space in the dynamic memory part of the computing memory; Use peer-to-peer (P2P) for transmission to transfer the SS-Table file from the computing memory storage module to the dynamic memory inside the computing memory; Pre-allocate a buffer for the merged data in the dynamic memory of the computing memory; Activate the computing unit in the computing memory to perform merge calculations; The host CPU receives the return parameters returned by the computing unit of the computing memory; The host CPU generates partial metadata of the target SS-Table file based on the parameters returned by the computing unit; Use P2P for transmission, and write the generated partial data of the SS-Table file from the dynamic memory of the computing memory back to the storage module of the computing memory.

[0034] In some embodiments, the host CPU further includes an intelligent merge operation trigger for dynamically triggering merge calculations according to the real-time load status of the system.

[0035] In some embodiments, dynamically triggering merge calculations according to the real-time load status of the system includes: Detect the real-time load status of the host CPU and the computing memory in the current system; According to the load status of the host CPU and the load status of the computing memory, determine the device type with a heavier workload and the priority level corresponding to the workload difference among the two types of devices, namely the host CPU and the computing memory, in the current system; According to the determined device type and priority level, trigger the merge trigger policy corresponding to the device and priority level; Estimate the actual computing device for analytical calculations and dynamically adjust the SS-Table file included in a single merge calculation; Generate corresponding merge calculations for the key-value database.

[0036] In some embodiments, Figure 1 The system in [[ ]] further includes a computing storage device intelligent dynamic scheduler for coordinating merge operations, task allocation, and resource allocation between the host CPU and the computing memory.

[0037] In some embodiments, coordinating merge operations, task allocation, and resource allocation between the host CPU and the computing memory includes: After receiving the merge calculation feedback information triggered by the key-value database, determine the computing overhead of the current calculation on the host CPU and the computing memory based on the merge level of the generated calculation, the number of SS-Tables included, and the file size characteristics; Compare the computing time of the host CPU and the computing memory and the impact on the overall system resources to determine the computing device; Add the corresponding merge calculation to the calculation queue of the corresponding computing device; Regularly detect the calculation queues of all devices and dynamically adjust the allocated computing devices and computing priorities according to the waiting time in the calculation queues.

[0038] In some embodiments, Figure 1 The system in [[ ]] further includes a key-value database parameter dynamic tuner, and the key-value database parameter dynamic tuner is used to regularly detect the workload of the entire system; Analyze the detected workload, and according to the tuning strategy, analyze the optimal strategy including memory buffer and background merge thread parameters in the current database; Adjust the parameters in the key-value database according to the analysis results.

[0039] Figure 2 It is an overall schematic diagram of a key-value database computing framework based on a compute storage device provided by an embodiment of the present disclosure, Figure 3 It is an overall architecture schematic diagram of a merged computing hardware accelerator kernel provided by an embodiment of the present disclosure. Combining Figure 2 and Figure 3 , Figure 1 The near-storage computing system of the key-value database based on the compute storage device in

[0040] From Figure 2 it can be seen that the hardware architecture of the present disclosure consists of three types of devices: a host processor (CPU), memory, and a compute memory. On the basis of maintaining a structure similar to the original Log-Structured Merge (LSM) tree, the compute memory replaces the traditional disk, and a dedicated compute accelerator is deployed on the acceleration device inside the compute memory. Under the design of this hardware architecture, in order to better utilize the newly added computing power of the compute storage device, an efficient intelligent merge operation trigger, a compute storage device intelligent dynamic scheduler, and a key-value database parameter dynamic tuner are also deployed on the host side.

[0041] For compaction computing, a hardware acceleration kernel is designed. This kernel is implemented based on a hardware description language and has a high degree of parallelism and a pipeline structure. In the key-value database, compaction computing mainly includes three parts: decoding, sorting, and encoding. The present disclosure accelerates compaction computing based on the data dependency relationship and data organization relationship of the merged computing. In the hardware acceleration kernel, the acceleration kernel includes three parts: a decoder, a sorter, and an encoder.

[0042] (1) The decoder is responsible for reading the SS-Table to be merged from the memory, then parsing out all key-value pairs from the file, and placing the key-value pairs in the FIFO buffer connected to the sorter. The operations included are: Judge whether the reading of the current data block is completed. If it is completed, read the next data block from the memory into the data block buffer stored on the BRAM; Each time, read the current data from the buffer and read the key-value pair metadata according to the current data; Read the key-value pair information according to the key-value pair metadata; Transmit the read key-value pair to the FIFO buffer; Update the read index according to the current key-value pair metadata.

[0043] (2) The sorter is mainly responsible for comparing the size relationship of key-value pairs. The sorter will take out key-value pairs from the corresponding decoder FIFO buffer according to the result of each comparison, and then select the key-value pair with the smallest key in the sorter according to the size of the key in the key-value pair, and transfer the corresponding key-value pair to the FIFO buffer connected to the encoder. The operations included are: Read a key-value pair from the FIFO buffer connected to the decoder; Separate the read key-value pair to obtain the user key (user-key) and timestamp in the key-value pair; Compare the sizes of all obtained user keys; Transfer the key-value pair with the smallest user key to the FIFO buffer connected to the encoder; Supplement key-value pairs from the decoder corresponding to the key-value pair with the smallest user key; (3) The encoder is responsible for re-encoding the existing key-value pairs into a file in the standard SS-Table format. The encoder takes out key-value pairs from the FIFO buffer connected to the sorter each time, and then generates an SS-Table file according to the file encoding rules and stores it in memory. The operations included are: Read a key-value pair from the FIFO buffer connected to the sorter; Determine the validity of the current key-value pair; Determine the encoding of the key-value in the current key-value pair according to the encoding status and the information of the previous key-value pair; Store the encoded key encoding in the buffer in memory; Store the encoded value encoding in the buffer in memory; Update the encoder status according to the current key-value pair.

[0044] Under this hardware accelerator design architecture, the decoder, sorter, and encoder will be started simultaneously after the kernel is activated. Each computing unit will determine its computing status according to the data status in the buffer connected to it. Once the required data is ready, the corresponding computing unit will be activated for computing, and the data flow relationship organized by the FIFO buffer can enable the three computing devices to effectively implement pipelined computing.

[0045] In a key-value database based on computational memory acceleration, the Compaction computing operation will generate a call to compute Compaction in the computational memory. The operations included in the call to the computational memory are: Statistical metadata of the SS-Table files that will participate in compaction; Map the SS-Table file space in the DRAM part of the computing memory; Use P2P for transmission to transfer the SS-Table file from the computing memory storage module to the DRAM inside the computing memory; Pre-allocate a buffer for the data generated by Compaction in the DRAM part of the computing memory; Activate the computing units in the computing memory to perform Compaction calculations; The host receives the return parameters returned by the computing units of the computing memory; The host side generates the metadata of the target SS-Table file part according to the parameters returned by the computing units; Use P2P for transmission to write the generated SS-Table file part data from the DRAM inside the computing memory back to the computing memory storage module.

[0046] In order to make full use of the computing potential of the computing storage device accelerator, the present disclosure deploys an efficient and intelligent merge operation trigger at the host side. This trigger will dynamically trigger Compaction calculations according to the real-time load status of the system. The operations included in the trigger are: Detect the current real-time load status of the system, including the usage rate and throughput rate and other status information of the two types of devices in the system; According to the CPU computing load and the load status of the computing memory, determine the device type with the heavier workload and the priority level corresponding to the workload difference among the two types of devices, namely the CPU and the computing memory, in the current system; According to the determined device type and priority level, trigger the Compaction trigger policy under the corresponding device and priority level; For the Compaction calculations generated under the dynamic policy, the trigger will estimate and analyze the actual computing devices for the calculations. If the computing device is the computing memory, the trigger will dynamically adjust the SS-Table files included in a single Compaction calculation; Generate corresponding Compaction calculations for the key-value database.

[0047] The present disclosure also includes a computing storage device intelligent dynamic scheduler, which is responsible for coordinating the merge operation task allocation and resource allocation between the host CPU and the computing memory. The operations included in the scheduler are: After receiving the Compaction calculations triggered by the key-value database, the scheduler will estimate the computing overheads of the current calculations on the CPU and the computing memory based on the merge level of the generated calculations, the number of SS-Tables included, and the file size characteristics; The scheduler determines the computing device by comparing the time taken by the CPU and the computing memory for calculations and the impact on the overall system resources; Add the corresponding merged calculation to the calculation queue of the corresponding computing device; The scheduler will regularly detect the calculation queues of all devices and dynamically adjust the allocated computing devices and calculation priorities according to the waiting time in the calculation queues.

[0048] To further improve the system performance, the present disclosure also provides a key-value database parameter dynamic tuner. The operations included in the tuner are: The dynamic tuner will regularly detect the workload of the entire system; Analyze the detected workload and analyze the optimal strategy for parameters including the memory buffer, background compaction thread, etc. in the current database according to the tuning strategy; Adjust the parameters in the key-value database according to the analysis results.

[0049] In a specific embodiment, in combination with algorithmic pseudocode, the specific calculation process of the acceleration kernel design based on computing storage devices, the efficient intelligent merge operation trigger, and the intelligent dynamic scheduler for computing storage devices proposed by the present invention is further described in detail.

[0050] 1. Acceleration Kernel Design for Computing Storage Devices The accelerator kernel designed based on computing storage devices proposed by the present invention has high throughput and low resource consumption, significantly improving the calculation speed of the merge operation. At the same time, it improves the overall write throughput of the key-value storage system, greatly reducing the write blocking and write compression problems that occur during the process, and ensuring the efficient operation of the key-value storage system.

[0051] Algorithm Strategy for Merge Operation Parallel Computing Unit

[0052] 2. Intelligent Merge Operation Trigger The intelligent merge operation trigger proposed by the present invention intelligently triggers the merge operation according to the current system workload and resource utilization rate. The strategy of the intelligent merge operation trigger is as follows: Intelligent Merge Operation Trigger

[0053] 3. Intelligent Dynamic Scheduler for Computing Storage Devices The intelligent dynamic scheduler for computing storage devices proposed by the present invention is based on the current system workload and the number of triggered compactations. The strategy of the intelligent merge operation trigger is as follows: Intelligent Merge Operation Trigger

[0054] The core idea of this disclosure is to offload part of the compaction operation to be executed in the compute memory. In previous experiments, the process and data flow dependencies of the compaction operation were deeply analyzed. Based on these analysis results, an accelerator core with high throughput and low resource consumption was designed. This core can be flexibly deployed in the compute memory to optimize data processing. In addition, a variety of compaction and scheduling strategies were tested using the physical compute memory, and a computing framework for the collaborative work of the compute storage device and the host device was carefully designed. This framework not only includes an efficient trigger compaction strategy but also covers an optimized scheduling strategy, aiming to achieve deep integration of computing and storage, make full use of the computing resources in the host, and improve the overall system performance.

[0055] The innovation of this disclosure is reflected in the following aspects: (1) Near-memory computing architecture: Leveraging the unique advantages of the compute storage device, a near-memory computing architecture is proposed for the compaction process in the key-value database. This architecture significantly reduces the data transfer overhead between the disk and the host memory, improves the throughput of the compaction calculation, and reduces the operation time delay.

[0056] (2) High-throughput accelerator core: For the compaction operation in the key-value database based on the LSM tree, an accelerator core with high throughput and low resource consumption is designed. This core is specifically optimized for the compaction operation to improve the overall performance.

[0057] (3) Collaborative work scheduling strategy: An efficient scheduling strategy for the collaborative work of the host side and the compute memory to accelerate the compaction operation, as well as a more intelligent compaction operation trigger strategy, are designed. These strategies ensure the optimal utilization of computing and storage resources and further improve the system performance.

[0058] (4) Optimized database setting parameters: This disclosure also provides a set of database setting parameters with better performance for the key-value database system integrated with the compute storage device. These parameters are carefully adjusted and optimized to give full play to the potential of the compute storage device and provide users with more efficient database performance.

[0059] Combined with Figure 4As shown in the figure, an embodiment of the present disclosure further provides a near-storage computing device 400 of a key-value database based on a compute-storage device, including a processor 404 and a memory 401. Optionally, the system may further include a communication interface 402 and a bus 403. Among them, the processor 404, the communication interface 402, and the memory 401 can complete communication with each other through the bus 403. The communication interface 402 can be used for information transmission. The processor 404 can call the logical instructions in the memory 401 to execute the method corresponding to the near-storage computing of the key-value database based on the compute-storage device in the above embodiment.

[0060] In addition, when the logical instructions in the above-mentioned memory 401 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium.

[0061] As a computer-readable storage medium, the memory 401 can be used to store software programs and computer-executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present disclosure. By running the program instructions / modules stored in the memory 401, the processor 404 executes functional applications and data processing, that is, implements the method corresponding to the near-storage computing of the key-value database based on the compute-storage device in the above embodiment.

[0062] The memory 401 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal device, etc. In addition, the memory 401 may include a high-speed random access memory and may also include a non-volatile memory.

[0063] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, and the computer-executable instructions are set to execute the method corresponding to the near-storage computing of the key-value database based on the compute-storage device.

[0064] The above computer-readable storage medium may be a transient computer-readable storage medium or a non-transient computer-readable storage medium.

Claims

1. A key-value database near storage computing system based on computing storage devices, characterized in that: The system includes a host CPU, a memory and a computing memory, wherein the computing memory is deployed with a memory and a computing accelerator, and the computing accelerator includes a decoder, a sequencer and an encoder; The decoder is used to read the SS-Table file to be merged from the memory, parse all the key-value pairs from the file, and place the key-value pairs in the FIFO buffer connected to the sorter; The sorter is used to compare the size relationship of the key-value pairs, and take out the key-value pairs from the corresponding decoder FIFO buffer according to the result of each comparison, select the key-value pair with the smallest key according to the size of the key in the key-value pair, and transmit the corresponding key-value pair to the FIFO buffer connected to the encoder; The encoder is used to take out the key-value pairs from the FIFO buffer connected to the sorter, and generate an SS-Table file according to the file encoding rule and store it in the memory.

2. The system according to claim 1, characterized in that The step of reading the SS-Table file to be merged from the memory, parsing all the key-value pairs from the file, and placing the key-value pairs in a FIFO buffer connected to the sorter includes: Determine whether the current data block reading is completed. If it is completed, read the next data block from the memory to the data block buffer stored in the BRAM; Read the current data from the buffer and read the key-value pair metadata based on the current data; Read key-value pair information according to key-value pair metadata; Transfer the read key-value pairs to the FIFO buffer; Updates the read index based on the current key-value pair metadata.

3. The system according to claim 1, characterized in that The comparing the size relationship of the key-value pairs, taking out the key-value pairs from the corresponding decoder FIFO buffer according to the result of each comparison, selecting the key-value pair with the smallest key according to the size of the key in the key-value pair, and transmitting the corresponding key-value pair to the FIFO buffer connected to the encoder, including: Read a key-value pair from the FIFO buffer connected to the decoder; Separate the read key-value pairs and obtain the user key value and timestamp in the key-value pairs; Compare all the obtained user key values; The key-value pair of the minimum user key value is transmitted to the FIFO buffer connected to the encoder; The decoder corresponding to the key-value pair of the minimum user key value supplements the key-value pair.

4. The system according to claim 1, characterized in that The method of taking out the key-value pairs from the FIFO buffer connected to the sorter and generating the SS-Table file according to the file encoding rule and storing it in the memory includes: Read key-value pairs from the FIFO buffer connected to the sorter; Determine the validity of the current key-value pair; Determine the encoding of the key value in the current key-value pair based on the encoding state and the information of the previous key-value pair; Store the encoded key code into the buffer in memory; Store the encoded value encoding into the buffer in memory; Updates the encoder state based on the current key-value pair.

5. The system according to claim 1, characterized in that The system is also used to: Count the metadata of the SS-Table files involved in the merge; Map the SS-Table file space in the dynamic memory portion of the computational memory; Use P2P to transfer the SS-Table file from the computing memory storage module to the dynamic memory inside the computing memory; Pre-allocating a buffer for merging generated data in a dynamic memory of a computing memory; Activate the calculation unit in the calculation memory to perform combined calculation; The host CPU receives the return parameter returned by the computing unit of the computing memory; The host CPU generates partial metadata of the target SS-Table file based on the parameters returned by the computing unit; Use P2P for transmission and write part of the data of the generated SS-Table file from the dynamic memory of the computing memory back to the storage module of the computing memory.

6. The system according to claim 1, characterized in that The host CPU also includes an intelligent merging operation trigger for dynamically triggering merging calculations according to the real-time load status of the system.

7. The system according to claim 6, characterized in that The dynamically triggering the merge calculation according to the real-time load status of the system includes: Detect the real-time load status of the host CPU and computing memory in the current system; According to the load status of the host CPU and the load status of the computing memory, determine the device type with heavier workload and the priority level corresponding to the workload difference between the two types of devices, the host CPU and the computing memory, in the current system; According to the determined device type and priority level, trigger the combined triggering strategy under the corresponding device and priority level; Estimate the actual computing equipment for analytical calculations and dynamically adjust the SS-Table files included in a single merge calculation; Generates corresponding merge calculations for the key-value database.

8. The system according to claim 1, characterized in that The system also includes an intelligent dynamic scheduler for computing storage devices, which is used to coordinate merging operations, task allocation, and resource allocation between the host CPU and the computing memory.

9. The system according to claim 8, characterized in that The coordination of merging operations, task allocation and resource allocation between the host CPU and the computing memory includes: After receiving the merge calculation feedback information triggered by the key-value database, the current calculation overhead on the host CPU and computing memory is determined based on the merge level of the generated calculation, the number of SS-Tables included, and the file size characteristics; Compare the host CPU and computing memory computing time and the impact on the overall system resources to determine the computing device; Add the corresponding merge calculation to the calculation queue of the corresponding computing device; The computing queues of all devices are checked regularly, and the allocated computing devices and computing priorities are dynamically adjusted according to the waiting time in the computing queues.

10. The system according to claim 1, characterized in that The system further comprises a key-value database parameter dynamic tuner, wherein the key-value database parameter dynamic tuner is used to periodically detect the workload of the entire system; Analyze the detected workload and analyze the optimal strategy for the current database including memory buffer and background merge thread parameters according to the tuning strategy; Adjust the parameters in the key-value database based on the analysis results.

Images