Distributed acceleration method and system for RocksDB storage system based on CSD master control

Abstract

The invention discloses a distributed acceleration method and system for a RocksDB storage system based on CSD master control. The method comprises the steps that CSD is deployed in the RocksDB storage system to serve as a computing node, a computing cluster is constructed, a Compaction process is stripped and migrated into the CSD, and an RPC channel is constructed based on the CSD; the compact instruction is directly written into a CSD memory, file meta-information is obtained through analysis, and a corresponding file is read; data needing to be calculated and a calculation instruction are sent to a calculation cluster of the CSD for calculation analysis, and calculation results of the far end are aggregated to the local; and migrating the data to the nand flash memory according to a storage instruction sent by the host side. According to the technical scheme, the CPU load of the host side is effectively reduced, the file data interaction efficiency is greatly improved, and the energy overhead and the time overhead of data migration interaction are greatly reduced.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a RocksDB storage system distributed acceleration method based on a CSD master control and a RocksDB storage system distributed acceleration system based on a CSD master control. Background Art

[0002] Traditional kv storage systems, such as RocksDB / leveldb, are kv persistent storage engines developed based on Log-Structured Merge (LSM) trees. They are embedded in the host-side application in the form of library components. Users write data through the kv interface, and the key and value are stored in the file through a certain data structure. The file system writes the data to the disk through the block device driver.

[0003] As an excellent embedded KV storage engine, RocksDB is widely used in many large distributed databases due to its excellent design and excellent data writing performance. However, as a KV storage engine, read amplification and write amplification of RocksDB have always been issues that affect its performance, especially in the process of importing external data.

[0004] Traditional kv storage solutions have the following problems:

[0005] (1) The performance jitter of the KV storage solution is serious

[0006] In the kv storage system based on lsm tree, if the total size of files in a level N exceeds a certain threshold, compaction will be performed to reduce space expansion. What compaction does is: merge multiple sst files in level N with files whose key ranges overlap in level N+1, and finally generate multiple new files and put them into level N+1.

[0007] When a large amount of overall disordered data is written to the new migration node, the key ranges of the sst files between levels will inevitably overlap, and the RocksDB on it will frequently perform a large number of compaction operations (rather than simple move) that require merging, sorting, and deduplication. This will inevitably occupy a large amount of IO, thus affecting the read and write performance, resulting in large performance fluctuations in the front-end business.

[0008] (2) The kv storage system consumes a lot of CPU resources

[0009] Compaction is a CPU-intensive operation. In the traditional architecture, Compaction and upper-layer services are coupled together, affecting each other and even blocking upper-layer writes (called WriteStall).

[0010] When flush / compaction cannot keep up with the write rate, RocksDB will reduce the write rate or even stop writing. When WriteStall occurs, cluster writes will become slower or even blocked, affecting the business.

[0011] (3) The computing cluster cost of KV storage is high

[0012] Traditionally, DB nodes serve both read and write requests and perform compaction, but the requirements for these two types of computing are completely different:

[0013] 1. Serving read and write requests is an online calculation, and the evaluation indicator is the latency of each request;

[0014] 2. Executing Compact is an offline calculation, and the evaluation indicator is the total throughput of all Compacts.

[0015] When these two types of calculations are executed concurrently, they will compete for resources. Even if the online calculation has a higher priority, it will eventually be dragged down by the offline calculation. Therefore, in actual applications, people generally dare not run the system load too high. For example, some businesses require that the load of online services should not exceed 30% during planning, but in fact it may be around 10% all year round.

[0016] On the other hand, in LSM engines such as RocksDB, the computing power consumed by the write operation itself does not exceed 10%, while the computing power consumed by the background threads (Flush & Compact) triggered by the write operation is the bulk of the computing power. According to the concept of RocksDBRemoteCompaction, the Compact is transferred to the Secondary Instance, so that the Secondary Instance only performs Compact and does not serve online requests. At this time, there is no need to add a dedicated compact server. As long as the CSD is plugged in to the local machine, it can provide a dedicated Compact service function to the computing cluster. Summary of the invention

[0017] In view of the above problems, the present invention provides a distributed acceleration method and system for a RocksDB storage system based on a CSD master control. While using the multi-partition function of the computational storage CSD to store data, a customized acceleration partition is used to directly store kv. When compaction is triggered, the compaction work is moved down to the chip without changing the business scenario logic, which effectively reduces the CPU load on the host side and greatly improves the interaction efficiency of file data. It can achieve the unification of large computing and high-throughput storage on the shortest path, greatly reducing the energy and time overhead of data migration interaction.

[0018] To achieve the above object, the present invention provides a RocksDB storage system distributed acceleration method based on CSD master control, comprising:

[0019] Deploy a preset number of computable storage CSDs as computing nodes in the RocksDB storage system to build a computing cluster.

[0020] The compaction process on the host side is separated and migrated to the CSD, and the RPC channel is built based on the CMB space in the CSD;

[0021] The compact instruction is directly written into the CSD memory, and the CSD memory parses the received compact instruction to obtain file meta information, and reads the corresponding file according to the file meta information;

[0022] Send the data and calculation instructions to be calculated to the computing cluster of the CSD for calculation analysis, and aggregate the remote calculation results to the local;

[0023] According to the storage instruction sent by the host side to the control end, the file system is called to migrate the data to the nand flash memory.

[0024] In the above technical solution, preferably, the RocksDB storage system accesses the lba on the CSD through the bluefs module.

[0025] In the above technical solution, preferably, for the host without the CSD deployed, a compact instruction is sent to the host agent with the CSD deployed to handle the compact process, and after the CSD agent completes the compact process, the compact result is returned to the local to continue to complete the subsequent process.

[0026] In the above technical solution, preferably, after the CSD is calculated, the remote lcompact indicator and the local dcompact indicator are aggregated.

[0027] In the above technical solution, preferably, the calculation instruction encapsulates an operation code of the calculation operation and the name of the data to be processed.

[0028] The present invention further proposes a RocksDB storage system distributed acceleration system based on a CSD master, which applies the RocksDB storage system distributed acceleration method based on a CSD master disclosed in any one of the above technical solutions, including:

[0029] The computing cluster deployment module is used to deploy a preset number of computable storage CSDs as computing nodes in the RocksDB storage system to build a computing cluster.

[0030] The computing decoupling and reconstruction module is used to separate the compaction process on the host side and migrate it to the CSD, and build an RPC channel based on the CMB space in the CSD;

[0031] A file parsing and reading module, used for writing a compact instruction directly into the CSD memory, the CSD memory parses the received compact instruction to obtain file meta information, and reads the corresponding file according to the file meta information;

[0032] The partition calculation aggregation module is used to send the data and calculation instructions that need to be calculated to the calculation cluster of the CSD for calculation analysis, and aggregate the remote calculation results to the local;

[0033] The file migration storage module is used to call the file system to migrate data to the NAND flash memory according to the storage instruction sent by the host side to the control end.

[0034] In the above technical solution, preferably, the RocksDB storage system accesses the lba on the CSD through the bluefs module.

[0035] In the above technical solution, preferably, for the host without the CSD deployed, a compact instruction is sent to the host agent with the CSD deployed to handle the compact process, and after the CSD agent completes the compact process, the compact result is returned to the local to continue to complete the subsequent process.

[0036] In the above technical solution, preferably, after the CSD is calculated, the remote lcompact indicator and the local dcompact indicator are aggregated.

[0037] In the above technical solution, preferably, the calculation instruction encapsulates an operation code of the calculation operation and the name of the data to be processed.

[0038] Compared with the prior art, the present invention has the following beneficial effects: while using the multi-partition function of the computational storage CSD to store data, the customized acceleration partition is used to directly store kv, and when compaction is triggered, the compaction work is moved down to the chip without changing the business scenario logic, which effectively reduces the CPU load on the host side and greatly improves the interaction efficiency of file data. It can achieve the unification of large computing and high-throughput storage on the shortest path, greatly reducing the energy and time overhead of data migration interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of a flow chart of a RocksDB storage system distributed acceleration method based on a CSD master control disclosed in an embodiment of the present invention;

[0040] Figure 2 A schematic diagram of the CSD internal architecture of a RocksDB storage system distributed acceleration method based on a CSD master control disclosed in an embodiment of the present invention;

[0041] Figure 3 A schematic diagram of the interaction between a host and a bluesfs module of a CSD disclosed in an embodiment of the present invention;

[0042] Figure 4 A schematic diagram of compaction interaction between hosts disclosed in an embodiment of the present invention;

[0043] Figure 5 A schematic diagram of a module of a RocksDB storage system distributed acceleration system based on a CSD master according to an embodiment of the present invention.

[0044] In the figure, the corresponding relationship between each component and the reference numeral is as follows:

[0045] 1. Computing cluster deployment module, 2. Computing decoupling and reconstruction module, 3. File parsing and reading module, 4. Partition computing aggregation module, 5. File migration and storage module. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0048] like Figure 1 and Figure 2 As shown, a RocksDB storage system distributed acceleration method based on CSD master control provided by the present invention includes:

[0049] Deploy a preset number of computable storage CSDs as computing nodes in the RocksDB storage system to build a computing cluster.

[0050] The compaction process on the host side is separated and migrated to the CSD, and the RPC channel is built based on the CMB space in the CSD;

[0051] The compact command is directly written into the CSD memory, and the CSD memory parses the received compact command to obtain file meta information, and reads the corresponding file according to the file meta information;

[0052] Send the data and computing instructions to be calculated to the CSD computing cluster for calculation analysis, and aggregate the remote calculation results to the local area;

[0053] According to the storage instruction sent by the host side to the control end, the file system is called to migrate the data to the nand flash memory.

[0054] In this implementation, while data is stored through the multi-partition function of the computational storage CSD, a customized acceleration partition is used to directly store kv. When compaction is triggered, the compaction work is moved down to the chip without changing the business scenario logic. On-chip resources are used to compress and merge data, which effectively reduces the CPU load on the host side and greatly improves the interaction efficiency of file data. It can achieve the unification of large computing and high-throughput storage on the shortest path, greatly reducing the energy and time overhead of data migration interaction.

[0055] Specifically, by deploying multiple CSD computing nodes on the local machine to build a computing cluster, multiple db storage nodes can share compact computing resources, improving resource utilization. The data and computing information to be calculated are sent to CSD for calculation, and then sent back from the remote end and aggregated locally, and special indicators are processed separately.

[0056] The compaction process of the original RocksDB storage system on the host side is stripped off. The host side issues compaction or data acceleration instructions and uses the computing cluster on the disk to process data. By migrating the compaction service to the CSD, there will be no background compaction service to affect read and write requests, which improves performance and stability. The CSD dedicated to compaction calculation can only optimize compaction.

[0057] Specifically, the host sends a custom NVMe command to the controller, which is divided into a storage command nvme io or a computing command nvme admin. If it is a storage io command, the system moves the data to the nand flash on the disk through a normal file system call. If it is a computing command, the nvme command encapsulates the opcode of the computing operation and the name of the data to be processed. After receiving the computing command, the storage controller calls the Storge link to forward the command to the computing cluster for processing, and then returns the corresponding result to the user interface layer.

[0058] The key-value interface is implemented in the firmware. The storage cluster forwards the key value data to the computation cluster through the storage link. The computation cluster then parses and stores the data.

[0059] Furthermore, EB1212 CSD supports CMB address space externally, builds RPC channel based on the CMB space, and writes compact acceleration command directly to CSD memory (i.e. EB1212 memory address). After receiving the compact command, EB1212 confirms whether it is the default table according to the add_column_edit parameter in the command, and parses the sst file information of this operation according to the manifest_vedit parameter in the command. EB1212 reads these files according to the file meta information. Since the host and EB1212 share nand storage, the file data read on EB1212 is consistent with the file data written from the host side.

[0060] like Figure 3As shown in the figure, there are bluefs modules and rocksdb modules on both the host and CSD, but the bluefs module on the CSD does not maintain its metadata. Instead, the file metadata is maintained through the bluefs module on the host, which simplifies the workload on the CSD and ensures metadata consistency.

[0061] In the above implementation, preferably, the RocksDB storage system does not support directly storing data on a bare SSD disk, and therefore accesses the lba on the CSD through the bluefs module.

[0062] In the above implementation, preferably, for a host without CSD deployed, a compact instruction is sent to a host agent deployed with CSD to handle the compact process, and after the CSD agent completes the compact process, the compact result is returned to the local to continue to complete the subsequent process.

[0063] like Figure 4 As shown in the figure, between the two hosts, host HOST-1 is not deployed with EB1212, while host HOST-2 is deployed with EB1212. Host HOST-1 proxies the compact request to HOST-2. The background compaction is an interaction of on-chip resources, and the efficiency of such interaction is far beyond the reach of traditional file-based databases. The unification of large computing and high-throughput storage on the shortest path greatly reduces the energy and time overhead of data migration interaction, thus realizing a chip-level solution that directly improves the application capabilities of the KV storage system.

[0064] In the above implementation, preferably, after CSD calculation, the remote lcompact indicator and the local dcompact indicator are aggregated, and the dcompact indicator on the DB side corresponds to the remote lcompact indicator.

[0065] Compared with traditional file reading and writing (interaction between the host and CSD), the compaction in this disk is the interaction of on-chip resources. The efficiency of such interaction is far beyond the reach of traditional file-based databases.

[0066] The unification of large-scale computing and high-throughput storage is achieved on the shortest path, greatly reducing the energy and time costs of data migration interaction.

[0067] like Figure 5 As shown, the present invention further proposes a RocksDB storage system distributed acceleration system based on a CSD master, which applies the RocksDB storage system distributed acceleration method based on a CSD master disclosed in any one of the above embodiments, including:

[0068] Computing cluster deployment module 1 is used to deploy a preset number of computable storage CSDs as computing nodes in the RocksDB storage system to build a computing cluster.

[0069] Computing decoupling and reconstruction module 2 is used to separate the compaction process on the host side and migrate it to the CSD, and build an RPC channel based on the CMB space in the CSD;

[0070] The file parsing and reading module 3 is used to write the compact instruction directly into the CSD memory, and the CSD memory parses the received compact instruction to obtain the file meta information, and reads the corresponding file according to the file meta information;

[0071] Partition calculation aggregation module 4 is used to send the data and calculation instructions to be calculated to the CSD computing cluster for calculation analysis, and aggregate the remote calculation results to the local;

[0072] The file migration storage module 5 is used to call the file system to migrate data to the nand flash memory according to the storage instruction sent by the host side to the control end.

[0073] In the above implementation, preferably, the RocksDB storage system accesses the lba on the CSD through the bluefs module.

[0074] In the above implementation, preferably, for a host without CSD deployed, a compact instruction is sent to a host agent deployed with CSD to handle the compact process, and after the CSD agent completes the compact process, the compact result is returned to the local to continue to complete the subsequent process.

[0075] In the above implementation, preferably, after CSD calculation, the remote lcompact indicator is aggregated with the local dcompact indicator.

[0076] In the above implementation, preferably, the calculation instruction encapsulates the operation code of the calculation operation and the name of the data to be processed.

[0077] According to the RocksDB storage system distributed acceleration system based on CSD master control disclosed in the above implementation, the functions to be implemented by each module thereof correspond to the steps of the RocksDB storage system distributed acceleration method based on CSD master control disclosed in the above implementation. During the implementation process, operations are performed with reference to the above implementation, which will not be repeated here.

[0078] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A distributed acceleration method for RocksDB storage system based on CSD master control, characterized in that: include: Deploy a preset number of computable storage CSDs as computing nodes in the RocksDB storage system to build a computing cluster. The compaction process on the host side is separated and migrated to the CSD, and the RPC channel is built based on the CMB space in the CSD; The compact instruction is directly written into the CSD memory, and the CSD memory parses the received compact instruction to obtain file meta information, and reads the corresponding file according to the file meta information; Send the data and calculation instructions to be calculated to the computing cluster of the CSD for calculation analysis, and aggregate the remote calculation results to the local; According to the storage instruction sent by the host side to the control end, the file system is called to migrate the data to the nand flash memory.

2. The RocksDB storage system distributed acceleration method based on CSD master control according to claim 1 is characterized in that: The RocksDB storage system accesses the lba on the CSD through the bluefs module.

3. The RocksDB storage system distributed acceleration method based on CSD master control according to claim 1 is characterized in that: For the host without the CSD deployed, the compact instruction is sent to the host agent with the CSD deployed to process the compact process, and after the CSD agent completes the compact process, the compact result is returned to the local to continue to complete the subsequent process.

4. The RocksDB storage system distributed acceleration method based on CSD master control according to claim 1 is characterized in that: After the CSD calculation, the remote lcompact index is aggregated with the local dcompact index.

5. The RocksDB storage system distributed acceleration method based on CSD master control according to claim 1 is characterized in that: The computing instruction encapsulates an operation code of the computing operation and a name of the data to be processed.

6. A RocksDB storage system distributed acceleration system based on CSD master control, characterized in that: The RocksDB storage system distributed acceleration method based on CSD master control as claimed in any one of claims 1 to 5 is applied, comprising: The computing cluster deployment module is used to deploy a preset number of computable storage CSDs as computing nodes in the RocksDB storage system to build a computing cluster. The computing decoupling and reconstruction module is used to separate the compaction process on the host side and migrate it to the CSD, and build an RPC channel based on the CMB space in the CSD; A file parsing and reading module, used for writing a compact instruction directly into the CSD memory, the CSD memory parses the received compact instruction to obtain file meta information, and reads the corresponding file according to the file meta information; The partition calculation aggregation module is used to send the data and calculation instructions that need to be calculated to the calculation cluster of the CSD for calculation analysis, and aggregate the remote calculation results to the local; The file migration storage module is used to call the file system to migrate data to the NAND flash memory according to the storage instruction sent by the host side to the control end.

7. The RocksDB storage system distributed acceleration system based on CSD master control according to claim 6, characterized in that: The RocksDB storage system accesses the lba on the CSD through the bluefs module.

8. The RocksDB storage system distributed acceleration system based on CSD master control according to claim 6, characterized in that: For the host without the CSD deployed, the compact instruction is sent to the host agent with the CSD deployed to process the compact process, and after the CSD agent completes the compact process, the compact result is returned to the local to continue to complete the subsequent process.

9. The RocksDB storage system distributed acceleration system based on CSD master control according to claim 6, characterized in that: After the CSD calculation, the remote lcompact index is aggregated with the local dcompact index.

10. The RocksDB storage system distributed acceleration system based on CSD master control according to claim 6, characterized in that: The computing instruction encapsulates an operation code of the computing operation and a name of the data to be processed.

Images