An I / O scheduling method and system for persistent KV storage based on LSM tree

Abstract

The present invention provides an I / O scheduling method and system for persistent KV storage based on LSM tree, belonging to the field of operating system storage, the scheduling method includes building a lightweight storage stack of an operating system based on kernel bypass, constructing a multi-layer scheduling model thereon, the first layer I / O dependency detection layer: detecting the dependency relationship formed between foreground and background I / O requests, and dynamically adjusting the priority of foreground and background I / O requests according to the dependency relationship; the second layer I / O scheduling layer: allocating I / O requests of different priorities to multi-core processing; the third layer I / O asynchronous assembly layer: ensuring that I / O requests are effectively propagated between the layers of the lightweight storage stack according to the priority level, and assembling the I / O requests in an asynchronous queue according to the priority level. The scheduling method described in the present invention can effectively reduce the system delay of persistent KV storage, improve the throughput of persistent KV storage, reduce the occurrence of I / O priority inversion problems, and optimize the performance overhead of persistent KV storage.

Description

Technical Field

[0001] The present invention belongs to the field of operating system storage, and relates to an I / O scheduling method and system for persistent KV storage based on an LSM tree. Technical Background

[0002] With the rapid advancement of the Internet, big data, and cloud computing technologies, traditional relational databases face performance limitations when processing large amounts of high-speed, unstructured data. To address this challenge, persistent KV storage (such as RocksDB, LevelDB, and Cassandra) significantly optimizes data read and write efficiency through its efficient key-value (KV) pair model. In addition, the introduction of new storage devices, especially NVMe SSDs, has shown significant performance improvements compared to traditional HDDs and SATA SSDs with old interfaces. This is due to the excellent parallel processing capabilities, higher throughput, and low latency of NVMe SSDs, which significantly improves the performance of persistent KV storage systems, not only enhancing data processing efficiency, but also expanding its application areas. Therefore, the integration of persistent KV storage and new storage hardware technologies provides strong support for meeting the performance and reliability requirements of growing data-intensive applications.

[0003] As the performance of new storage devices continues to improve, the performance overhead challenges faced by persistent KV storage systems have become increasingly apparent, becoming the main bottleneck limiting the performance improvement of data-intensive environments. To meet this challenge, it is urgent to optimize and adjust the storage architecture in a refined manner based on the characteristics of persistent KV storage systems. This process aims to make full use of the high-performance characteristics of new storage hardware such as NVMe SSDs to optimize data processing speed and improve the overall reliability of the system, thereby significantly improving the performance efficiency of the storage system. This requires in-depth analysis of the interaction mechanism between the persistent KV storage system and storage hardware, and the formulation of a strategy that can meet the needs of efficient data access while ensuring data integrity and system stability.

[0004] The performance improvement of persistent KV storage comes from the introduction of LSM tree (Log-Structured Merge Tree). According to the design of LSM tree, I / O requests are divided into foreground I / O requests and background I / O requests. Foreground I / O requests, such as read and write operations; background I / O requests, such as refresh and compression operations. Since there is a dependency between foreground I / O and background I / O, and foreground and background I / O requests jointly occupy the storage stack, storage bandwidth contention occurs, which in turn causes an increase in I / O request latency and a decrease in throughput. Therefore, it is necessary to better prioritize foreground and background I / O, and on this basis, better schedule foreground and background I / O to reduce system latency. Currently, persistent key-value storage systems use static priority to divide I / O requests. In intensive read-write scenarios, there is a lack of real-time analysis of I / O request relationships and dynamic adjustment of I / O priorities. Therefore, there is an urgent need to explore new optimization methods for the scheduling strategy of the storage stack in order to better dynamically adjust the foreground and background I / O requests to adapt to the rapidly changing hardware technology and data processing requirements that are constantly evolving, thereby ensuring the continuous improvement of storage system performance. Summary of the invention

[0005] The present invention provides an I / O scheduling method for persistent KV storage based on LSM tree, so as to effectively reduce the system delay of persistent KV storage, increase the throughput of persistent KV storage, reduce the occurrence of I / O priority inversion problem, and optimize the performance overhead of persistent KV storage.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An I / O scheduling method for persistent KV storage based on LSM tree includes the following steps:

[0008] Based on the operating system storage stack built with kernel bypass, a multi-layer scheduling model is constructed, including the first layer I / O dependency detection layer, the second layer I / O scheduling layer and the third layer I / O asynchronous assembly layer;

[0009] The first layer, the I / O dependency detection layer, distinguishes the foreground and background I / O requests in the I / O request pool, analyzes the types of foreground and background I / O requests and their internal connections by distinguishing the intensive read and write types, determines the dependency between the foreground and background I / O requests, and dynamically adjusts the priority of the foreground and background I / O requests based on the dependency.

[0010] The second layer, I / O scheduling layer: allocates I / O requests of different priorities to multiple cores for processing according to different dependencies, and allocates I / O requests with dependencies to the same core for processing;

[0011] The third layer, I / O asynchronous assembly layer: ensures that I / O requests are effectively propagated between the layers of the storage stack (file system layer and block layer) according to their priority, assembles asynchronous I / O queues in the order of high-priority I / O requests first and low-priority I / O requests later, and finally sends the I / O queues to the storage hardware.

[0012] Furthermore, the first I / O dependency detection layer detects the dependency relationship formed between the foreground and background I / O requests using the following steps:

[0013] Get the information of foreground and background I / O requests, including the type of foreground request and the type of background request;

[0014] Based on the dynamic changes in memory and hard disk resources and the specific types of I / O requests, determine whether there is a dependency between I / O requests.

[0015] Furthermore, judging whether there is a dependency relationship between the I / O requests according to the dynamic changes of memory and hard disk resources and the specific types of the I / O requests includes:

[0016] Determine the dependency between I / O requests by the resource usage between I / O requests;

[0017] Parse the metadata of the I / O requests and determine the dependencies between the I / O requests based on the order in which the I / O requests are executed.

[0018] Furthermore, dynamically adjusting the priorities of foreground and background I / O requests according to the dependency relationship includes:

[0019] By parsing the metadata of the I / O request, the foreground I / O request and the background I / O request as well as the I / O bandwidth usage information are obtained, and the information of the multi-level compression request is stored in the compression relationship table;

[0020] By monitoring the number of read and write I / O requests over a period of time, determine whether the current I / O type is read-intensive or write-intensive:

[0021] If it is write-intensive, first query the write status of the SSTable table at the L0 level in the hard disk. If the data in the SSTable table at the L0 level is full and the L0~L1 layer is being compressed, increase the priority of the compression request at the L0~L1 level to the same as the priority of the write request. At the same time, according to the compression relationship queried in the compression relationship table, stop the cascade compression related to the L0~L1 layer compression, and continue to perform the cascade compression operation when the L0~L1 layer compression is completed; when it is queried that the SSTables table at the L0 level in the hard disk is not full, according to the bandwidth flow information of the I / O request, due to the L N To L N+1Layered compression takes up a lot of bandwidth, so increasing L N To L N+1 Layer-level compression, adjusting it to the same priority as write requests;

[0022] If it is read-intensive, when there is a read I / O request, the location of the data to be read is queried. If the data exists in the memory, the data is read directly without adjusting the I / O request priority. If the data exists in the hard disk, the priority of the compression operation where the data is located is increased and adjusted to the same priority as the read operation.

[0023] Furthermore, before dynamically adjusting the priorities of foreground and background I / O requests, the initial I / O request priorities are set as follows: Write / Read>Flush>L0~L1 low-level compression>L N ~L N+1 High level of compression.

[0024] Furthermore, in the second-layer I / O scheduling layer, after determining the dependencies of I / O requests and dynamically prioritizing the I / O requests, I / O requests with close dependencies are allocated to the same core for processing based on the dependencies between the I / O requests. The throughput of persistent KV storage is increased by performing multi-core processing on the I / O requests, and a tag TAG is added to the I / O requests for use when the I / O requests are propagated between storage stack layers.

[0025] Furthermore, in the third-layer I / O asynchronous assembly layer, an I / O request controller is created. When the I / O request passes through the file system layer and the block layer in the storage stack, the TAG information of the I / O request is analyzed, and the I / O request with high priority is given priority to pass through the storage stack; and the I / O queue is assembled according to the priority, with the high-priority I / O request in front and the low-priority I / O request in the back, and the assembled I / O queue is sent to the data path and finally reaches the storage device.

[0026] An I / O scheduling system for persistent KV storage based on an LSM tree includes a multi-layer scheduling model constructed on the basis of a storage stack of an operating system, wherein the multi-layer scheduling model includes a first-layer I / O dependency detection layer, a second-layer I / O scheduling layer, and a third-layer I / O asynchronous assembly layer; the first-layer I / O dependency detection layer detects the dependency relationship formed between foreground and background I / O requests, and dynamically adjusts the priority of foreground and background I / O requests according to the dependency relationship; the second-layer I / O scheduling layer allocates I / O requests of different priorities to multi-core processing, and allocates I / O requests with dependencies to the same core for processing; the third-layer I / O asynchronous assembly layer ensures that I / O requests are effectively propagated between layers of the storage stack according to priority, assembles I / O requests into asynchronous I / O queues according to priority, and sends the I / O queues to storage hardware.

[0027] The beneficial effects of the present invention are as follows:

[0028] Based on the above-mentioned scheme of the present invention, the first layer I / O dependency detection layer determines the dependency relationship formed between the foreground and background I / O requests by analyzing the types and generation relationships of foreground and background I / O requests, and dynamically adjusts the priority of the foreground and background I / O requests according to the dependency relationship; the second layer I / O scheduling layer: allocates I / O requests of different priorities to multi-core processing, and allocates I / O requests with dependencies to the same core for processing; the third layer I / O asynchronous assembly layer: ensures that I / O requests are effectively propagated between the layers of the storage stack according to the priority level, and performs asynchronous queue assembly; the use of a multi-layer scheduling method can effectively reduce the system delay of persistent KV storage, increase the throughput of persistent KV storage, reduce the occurrence of I / O priority inversion problems, and optimize the performance overhead of persistent KV storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is an architectural diagram of I / O scheduling for persistent KV storage based on LSM tree;

[0030] Figure 2 It is the first layer I / O dependency detection layer that analyzes the dependency between I / O requests;

[0031] Figure 3 It is the I / O scheduling strategy in the second layer I / O scheduling layer;

[0032] Figure 4 It is a diagram of I / O priority propagation and queue assembly in the third layer I / O asynchronous assembly layer. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below through specific embodiments and drawings.

[0034] The present invention provides an I / O scheduling method for persistent KV storage based on LSM tree, such as Figure 1 As shown, based on the storage stack of the operating system built with kernel bypass, a multi-layer scheduling model is constructed, which includes the first layer I / O dependency detection layer, the second layer I / O scheduling layer and the third layer I / O asynchronous assembly layer.

[0035] In the first layer I / O dependency detection layer, the dependency relationship between I / O requests is analyzed, such as Figure 2 As shown. First, establish an I / O request collector to obtain detailed information about foreground and background I / O requests, including the type of foreground request (such as read or write) and the type of background request (such as flush or compact, and the specific level of compaction). Set the initial I / O request priority to: Write / Read (read / write) > Flush (refresh) > Level 0 (L0) ~ Level 1 (L1) Compaction (low-level compaction) > Level N (L N )~level N+1(L N+1 )Compaction (high-level compression). Then, first determine the intensive read / write scenario. According to the characteristics of the LSM tree, special attention should be paid to the L0 level. When the L0 level is full or reaches the threshold, L0~L1 compression is triggered, and the priority of L0~L1 compression is dynamically increased to the highest. Unlike the static division in the prior art, this can avoid compression requests from occupying a large amount of system bandwidth, which helps to improve system performance. When L0 is not full, it is necessary to query the intrinsic connection (ie, dependency) between compression requests to find the I / O requests that affect the bandwidth, which are called critical I / O requests, and then increase the priority of critical I / O requests.

[0036] like Figure 2 As shown, the present invention determines whether there is a dependency relationship between I / O requests according to the dynamic changes of memory and hard disk resources and the specific types of I / O requests. The specific determination method is:

[0037] 1) Determine the dependency between I / O requests by the resource occupancy between I / O requests. For example, a refresh request writes data in memory to the SSTable in the L0 layer. During the data writing process, in order to optimize the space of the storage device, the SSTable in the L0 layer needs to be merged and compressed into the SSTable in the L1 layer. Therefore, there is a dependency between the refresh and L0-L1 compression requests.

[0038] 2) By parsing the metadata of the I / O requests, the dependencies between the I / O requests are determined by the order in which the I / O requests are executed. For example, the dependencies between cascade compression requests, I / O request 1 is L0-L1 layer compression, I / O request 2 is L1-L2 layer compression, in this case, there is a dependency between I / O request 1 and I / O request 2.

[0039] The method of dynamically dividing I / O priorities of the present invention is:

[0040] 1) By parsing the metadata of the I / O request, detailed information such as the foreground I / O request (such as read or write) and the background I / O request (such as refresh or compression, and which level of compression) and I / O bandwidth usage are obtained, and the information of the multi-level compression request is stored in the compression relationship table.

[0041] 2) Determine whether the current I / O type is read-intensive or write-intensive by monitoring the number of read and write I / O requests over a period of time:

[0042] If it is write-intensive, first query the write status of the L0-level SSTable table in the hard disk. If the data in the L0-level SSTable table is full and the L0-L1 layer is being compressed, increase the priority of the L0-L1 layer compression request to the same as the write request priority. At the same time, according to the compression relationship queried in the compression relationship table, stop the cascade compression related to the L0-L1 layer compression. When the L0-L1 layer compression is completed, continue to perform the cascade compression operation. When it is queried that the L0-level SSTables table in the hard disk is not full, according to the bandwidth flow information of the I / O request, due to the L N To L N+1 Layered compression takes up a lot of bandwidth, so increasing L N To L N+1 Level-by-level compression, giving it the same priority as write requests.

[0043] If it is read-intensive, the system queries the location of the data to be read through the Bloom filter (in the LSM tree, the Bloom filter is usually used in conjunction with the SSTable to improve query performance). When there is a read I / O request, the location of the data to be read is queried. If the data exists in the memory, the data can be read directly without occupying bandwidth. Therefore, there is no need to adjust the I / O request priority. If the data exists on the hard disk, the priority of the compression operation where the data is located is increased to be the same as the priority of the read operation.

[0044] The second layer I / O scheduling layer is as follows Figure 3As shown. In the second layer I / O scheduling layer, after determining the dependencies of I / O requests and dynamically dividing the priorities of I / O requests, I / O requests with close dependencies are divided into the same core for processing according to the dependencies between I / O requests. Multi-core processing is used to process I / O requests, which helps to increase the throughput of persistent KV storage and reduce system latency. At the same time, a tag (TAG) is added to the I / O request, which is used when the I / O request propagates between storage stack layers.

[0045] The third layer I / O asynchronous assembly layer is as follows Figure 4 As shown. In the third layer I / O asynchronous assembly layer, an I / O request controller is created. When the I / O request passes through the stack structure such as the file system layer and the block layer in the storage stack, the TAG information of the I / O request is analyzed, and the I / O request with high priority is allocated first (that is, the high priority I / O request passes through the storage stack first), which can prevent the priority inversion of the I / O request. The I / O queue is assembled according to the priority, with the high priority I / O request in front and the low priority I / O request in the back. The assembled I / O queue is sent to the data path and finally reaches the new storage device represented by NVMe SSD. This scheduling method can effectively reduce the system delay of persistent KV storage, increase the throughput of persistent KV storage, reduce the occurrence of I / O request priority inversion problem, and optimize the performance overhead of persistent KV storage.

[0046] Another embodiment of the present invention provides an I / O scheduling system for persistent KV storage based on an LSM tree, which includes a multi-layer scheduling model constructed on the basis of a storage stack of an operating system, wherein the multi-layer scheduling model includes a first-layer I / O dependency detection layer, a second-layer I / O scheduling layer, and a third-layer I / O asynchronous assembly layer; the first-layer I / O dependency detection layer detects the dependency relationship formed between foreground and background I / O requests, and dynamically adjusts the priority of foreground and background I / O requests according to the dependency relationship; the second-layer I / O scheduling layer allocates I / O requests of different priorities to multi-core processing, and allocates I / O requests with dependencies to the same core for processing; the third-layer I / O asynchronous assembly layer ensures that I / O requests are effectively propagated between the layers of the storage stack according to the priority level, assembles the I / O requests into asynchronous I / O queues according to the priority level, and sends the I / O queues to the storage hardware.

[0047] The specific implementation process of the multi-layer scheduling model refers to the above description of the method of the present invention.

[0048] Another embodiment of the present invention provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0049] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the steps of the method of the present invention are implemented.

[0050] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and implement it accordingly. It can be understood by those skilled in the art that various replacements, changes and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the contents disclosed in the embodiments of this specification, and the scope of protection of the present invention shall be subject to the scope defined in the claims.

Claims

1. An I / O scheduling method for persistent KV storage based on LSM tree, characterized in that: The following steps are involved: A multi-layer scheduling model is constructed based on the storage stack of the operating system, including the first layer I / O dependency detection layer, the second layer I / O scheduling layer and the third layer I / O asynchronous assembly layer; The first I / O dependency detection layer detects the dependency relationship between the foreground and background I / O requests, and dynamically adjusts the priority of the foreground and background I / O requests based on the dependency relationship; The second layer, the I / O scheduling layer, allocates I / O requests of different priorities to multiple cores for processing, and allocates I / O requests with dependencies to the same core for processing; The third layer, the I / O asynchronous assembly layer, ensures that I / O requests are effectively propagated among the various layers of the storage stack according to their priorities, assembles I / O requests into asynchronous I / O queues according to their priorities, and sends the I / O queues to the storage hardware. The dynamically adjusting the priorities of the foreground and background I / O requests according to the dependency relationship includes: By parsing the metadata of the I / O request, the foreground I / O request and the background I / O request as well as the I / O bandwidth usage information are obtained, and the information of the multi-level compression request is stored in the compression relationship table; By monitoring the number of read and write I / O requests over a period of time, determine whether the current I / O type is read-intensive or write-intensive: If it is write-intensive, first query the write status of the SSTable table at the L0 level in the hard disk. If the data in the SSTable table at the L0 level is full and the L0~L1 layer is being compressed, increase the priority of the compression request at the L0~L1 level to the same as the priority of the write request. At the same time, according to the compression relationship queried in the compression relationship table, stop the cascade compression related to the L0~L1 layer compression. When the L0~L1 layer compression is completed, continue to perform the cascade compression operation; when it is queried that the SSTables table at the L0 level in the hard disk is not full, increase the L N To L N+1 The priority of the level's compression is adjusted to the same priority as the write request; If it is read-intensive, when there is a read I / O request, the location of the data to be read is queried. If the data exists in the memory, the data is read directly without adjusting the I / O request priority. If the data exists in the hard disk, the priority of the compression operation where the data is located is increased and adjusted to the same priority as the read operation.

2. The method according to claim 1, characterized in that: The detection of the dependency relationship between the foreground and background I / O requests includes: Get the information of foreground and background I / O requests, including the type of foreground request and the type of background request; Based on the dynamic changes in memory and hard disk resources and the specific types of I / O requests, determine whether there is a dependency between I / O requests.

3. The method according to claim 2, characterized in that The determining whether there is a dependency relationship between the I / O requests according to the dynamic changes of memory and hard disk resources and the specific types of the I / O requests includes: Determine the dependency between I / O requests by the resource usage between I / O requests; Parse the metadata of the I / O requests and determine the dependencies between the I / O requests based on the order in which the I / O requests are executed.

4. The method according to claim 1, characterized in that Before dynamically adjusting the priority of foreground and background I / O requests, set the initial I / O request priority to: Write / Read> Flush> L0~ L1 low-level compression> L N ~L N+1 High level of compression.

5. The method according to claim 1, characterized in that: In the second-layer I / O scheduling layer, after determining the dependencies of I / O requests and dynamically prioritizing the I / O requests, I / O requests with close dependencies are divided into the same core for processing according to the dependencies between the I / O requests. The throughput of persistent KV storage is increased by multi-core processing of I / O requests, and a tag TAG is added to the I / O request for use when the I / O request propagates between storage stack layers.

6. The method according to claim 5, characterized in that In the third-layer I / O asynchronous assembly layer, an I / O request controller is created. When an I / O request passes through the file system layer and the block layer in the storage stack, the TAG information of the I / O request is analyzed, and the I / O request with high priority is given priority to pass through the storage stack; and the I / O queue is assembled according to the priority, with the high-priority I / O request in front and the low-priority I / O request in the back, and the assembled I / O queue is sent to the data path and finally reaches the storage device.

7. An I / O scheduling system for persistent KV storage based on LSM tree using the method described in any one of claims 1 to 6, characterized in that: The invention comprises a multi-layer scheduling model constructed on the basis of the storage stack of the operating system, wherein the multi-layer scheduling model comprises a first-layer I / O dependency detection layer, a second-layer I / O scheduling layer and a third-layer I / O asynchronous assembly layer; the first-layer I / O dependency detection layer detects the dependency relationship formed between the foreground and background I / O requests, and dynamically adjusts the priority of the foreground and background I / O requests according to the dependency relationship; the second-layer I / O scheduling layer allocates I / O requests of different priorities to multi-core processing, and allocates I / O requests with dependencies to the same core for processing; The third layer I / O asynchronous assembly layer ensures that I / O requests are effectively propagated among the layers of the storage stack according to their priorities, assembles I / O requests into asynchronous I / O queues according to their priorities, and sends the I / O queues to the storage hardware.

8. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 6 is implemented.

Images