Hotspot-aware log-structured merge tree read and write performance optimization method and related equipment

Abstract

The present invention discloses a hotspot-aware log-structured merge tree read-write performance optimization method and related equipment, belonging to the field of data storage technology. The method includes: maintaining a coarse-grained cache and a fine-grained cache in memory, caching the most recently accessed data in the system with data blocks and key-value pairs as granularity, respectively; the coarse-grained cache is a read-only cache used for service range query operations; the fine-grained cache is a read-write hybrid cache used for service point query operations and write operations; a hotspot-aware data merging method is introduced on the hard disk, a data heat calculation method is designed and implemented, and the execution of internal data merge operations in the log-structured merge tree is scheduled based on data access heat, while invalid data involved in the merge is quickly pruned, and after the merge is completed, hotter new data blocks are pre-fetched into the coarse-grained cache. The present invention can improve the cache hit rate, shorten the read and write path, and optimize the read and write performance of the log-structured merge tree.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and more specifically, relates to a hotspot-aware log structure merge tree read and write performance optimization method and related equipment. Background Art

[0002] With the rapid development of information technology and the explosive growth of data volume, massive amounts of data have been generated in various industries and fields, and the performance requirements for storage systems have also been further increased. Traditional data storage systems often use relatively fixed data models to organize and manage data. Data models are usually established based on some simple data scenarios, such as using simple logical two-dimensional tables to display data. Therefore, they cannot cope with the trend of diversified data forms and high-performance concurrent reading and writing requirements in the big data era. Unstructured data storage systems have come into being.

[0003] Unstructured databases based on log-structured merge trees (LSM-Trees) have become the foundation of modern storage systems in recent years, providing critical data storage services in the form of key-value stores. LSM-Tree storage systems take into account the fact that sequential read and write performance far outperforms random read and write performance on storage devices. They achieve high write performance by utilizing memory-delayed batch writes, making them widely used in write-intensive applications.

[0004] While log-structured merge trees offer excellent write performance, their read performance is lower than their write performance due to the need for continuous backend merge operations to promptly delete expired data and maintain data order on storage devices. This makes them inadequate for wider mixed read-write scenarios. Furthermore, today's application workloads often contain data hotspots, meaning a small amount of data accounts for the majority of access requests. Existing log-structured merge trees don't account for these hotspot characteristics. Furthermore, merge operations cause hotspot data to be repeatedly read and written on storage devices, competing with frontend user requests for internal server resources, resulting in decreased read and write performance.

[0005] In general, the read and write performance of existing storage systems based on log-structured merge trees needs to be further improved. Summary of the Invention

[0006] In response to the defects and improvement needs of the existing technology, the present invention provides a hotspot-aware log-structured merge tree read and write performance optimization method and related equipment, the purpose of which is to optimize the read and write performance of the log-structured merge tree.

[0007] To achieve the above objectives, according to one aspect of the present invention, a hotspot-aware log-structured merge tree read and write performance optimization method is provided, comprising:

[0008] Maintain a coarse-grained cache and a fine-grained cache in memory, caching the most recently accessed data in the system at the granularity of data blocks and key-value pairs respectively;

[0009] Furthermore, the point query operation for querying the target key-value pair includes:

[0010] (R1) Query the target key-value pair in the fine-grained cache. If the query is successful, go to step (R4); otherwise, go to step (R2);

[0011] (R2) Access each component in the order of write buffer, read-only write buffer, coarse-grained cache, and log-structured merge tree until the target key-value pair is found. If the target key-value pair is not found in all components, the query operation ends; otherwise, go to step (R3);

[0012] (R3) Cache the queried key-value pair into the fine-grained cache; if the target key-value pair is queried in the log-structured merge tree, cache the data block containing the target key-value pair into the coarse-grained cache;

[0013] (R4) Return the key-value pair found, and the query operation ends.

[0014] Furthermore, a dirty data queue is maintained in the fine-grained cache to record dirty data generated by updates in the fine-grained cache;

[0015] Furthermore, the execution of the write operation includes:

[0016] (W1) Write the key-value pair to be written into the write-before log;

[0017] (W2) Determine whether the key-value pair to be written is located in the fine-grained cache. If so, write the key-value pair to be written into the fine-grained cache by in-place updating, record the data to be written into the dirty data queue, set the corresponding dirty data flag, and then proceed to step (W3); otherwise, write the key-value pair to be written into the write buffer, and then proceed to step (W3);

[0018] (W3) Returns the update result and the write operation ends.

[0019] Furthermore, a range query operation for querying key-value pairs within a specified range includes:

[0020] (S1) If the dirty data queue is not empty, all dirty data in the fine-grained cache are read and sorted in batches, and then written into the write buffer in batches, and then the process proceeds to step (S2); otherwise, the process proceeds directly to step (S2);

[0021] (S2) reading key-value pairs within a specified range from the write buffer and the read-only write buffer, and determining a data block in the daily structure merge tree whose key-value range overlaps with the specified range; if the data block has been cached in the coarse-grained cache, reading the corresponding data block from the coarse-grained cache; otherwise, reading the data block from the log structure merge tree and caching it in the coarse-grained cache; reading key-value pairs within the specified range from the read data block;

[0022] (S4) The read key-value pairs are merged and the merged key-value pair set is returned, and the range query operation ends.

[0023] Furthermore, the coarse-grained cache is divided into a plurality of first partitions, each of which uses a hash table to manage data blocks; the fine-grained cache is divided into a plurality of second partitions, each of which uses a hash table to manage key-value pairs.

[0024] Furthermore, the merge operation of the log-structured merge tree includes:

[0025] Determine the level in the log-structured merge tree where a merge needs to be initiated, calculate the popularity of each data table in that level, and select the data table with the lowest popularity to initiate the merge operation;

[0026] The heat of a data table is the sum of the heats of all data blocks in the data table; the heat of a data block is the sum of the heats of all key-value pairs in the data block.

[0027] Furthermore, in the log structure merge tree, the heat calculation method of the key-value pair includes:

[0028] Determine whether the key-value pair is in the fine-grained cache. If so, set its read heat to s r Otherwise, set its read heat to 0; s r >0;

[0029] Determine whether the key-value pair has been discarded in the previous merge operation. If so, set its write heat to s w , otherwise, set its write heat to 0;s w >0;

[0030] The sum of the read heat and write heat of the key-value pair is taken as the heat of the key-value pair.

[0031] Furthermore, the merge operation of the log-structured merge tree also includes:

[0032] For the new data blocks generated during the merging process, the heat and average heat of each data block are calculated, and the data blocks with heat greater than the average heat are selected to replace the data blocks in the coarse-grained cache that have become invalid due to the merging.

[0033] Furthermore, the merge operation of the log-structured merge tree also includes:

[0034] For key-value pairs that need to be merged in the log-structured merge tree, if a newer version of the data has been cached in the fine-grained cache, write buffer, or read-only write buffer, the key-value pair is discarded.

[0035] According to another aspect of the present invention, a storage controller is provided, comprising: a computer-readable storage medium and a processor; the computer-readable storage medium stores a computer program; the processor is used to read the computer program in the computer-readable storage medium and execute the above-mentioned hotspot-aware log structure merge tree read and write performance optimization method provided by the present invention.

[0036] According to another aspect of the present invention, a storage system based on a log-structured merge tree is provided, comprising: a memory, a storage device, and the above-mentioned storage controller provided by the present invention.

[0037] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0038] (1) The hotspot-aware log-structured merge tree read-write performance optimization method and related equipment provided by the present invention maintain a dual-granularity cache in memory, namely a coarse-grained cache and a fine-grained cache, which respectively cache data at the granularity of data blocks and key-value pairs. The fine-grained cache is used to cache the most recently accessed key-value pairs, and as the first component of the service point query request, based on the temporal locality of key-value pair access, the point query operation will have a high probability of hitting the fine-grained cache without further access to other components. Therefore, the present invention can fully consider the hotspot characteristics of the data load, effectively shorten the access path of the point query operation, and improve the execution performance of the point query operation.

[0039] (2) The hotspot-aware log structure merge tree read and write performance optimization method and related equipment provided by the present invention, in its preferred embodiment, the fine-grained cache maintained in the memory also serves as the first component of the service write operation, which can fully consider the hotspot characteristics of the data load, effectively shorten the access path of the write operation, and improve the execution performance of the write operation.

[0040] (3) The hotspot-aware log structure merge tree read and write performance optimization method and related equipment provided by the present invention, in its preferred embodiment, both the coarse-grained cache and the fine-grained cache manage data by combining partitions with hash tables. Specifically, the cache is divided into multiple partitions, and a hash table is used to manage data in each partition, thereby effectively improving the concurrent read and write capabilities of the cache and further improving the read and write performance.

[0041] (4) The hotspot-aware log-structured merge tree read-write performance optimization method and related equipment provided by the present invention, in its preferred embodiment, when the merge operation executed in the background selects the data table for initiating the merge, the heat of each data table in the current layer of the log-structured merge tree will be calculated first, and the data table with the smallest heat will be selected as the initiator of the merge operation, thereby ensuring that the hotter data tables are kept in the upper layer of the log-structured merge tree as much as possible to quickly absorb write requests or serve read requests, shorten the read and write path of the log-structured merge tree, and further improve the read and write performance; further preferably, when calculating the heat of the key-value pairs in the data table, fine-grained cache will be combined to and the data discard records of the previous merge operation to identify the hot and cold data of a single key-value pair. Specifically, considering that in the present invention, the data in the fine-grained cache is only inserted into the fine-grained cache during the point query operation, the present invention identifies the data in the fine-grained cache as hot read data, and if the key-value pair data has been discarded in the previous merge operation, it means that the data has been written multiple times. Therefore, the present invention identifies the data discarded in the previous merge operation as hot write data, thereby improving the hot spot perception ability of the log structure merge tree based on the data management mechanism, and accurately and efficiently calculating the heat of key-value pairs, data blocks and data tables.

[0042] (5) The hotspot-aware log structure merge tree read and write performance optimization method and related equipment provided by the present invention, in its preferred embodiment, will select a data block with higher popularity from the newly generated data blocks during the merge operation to replace the data block in the coarse-grained cache that has failed due to the merge operation, thereby realizing the pre-fetching of hot data, thereby reducing the cache failure caused by the merge operation in the coarse-grained cache and improving the cache hit rate.

[0043] (6) The hotspot-aware log structure merge tree read and write performance optimization method and related equipment provided by the present invention, in its preferred embodiment, will identify expired and invalid data during the merge operation and directly discard it, thereby reducing the repeated reading and writing of invalid data on the storage device, reducing read and write amplification and space amplification, improving read and write performance, and extending the service life of the storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 A schematic diagram of a hotspot-aware log-structured merge tree read and write performance optimization method provided by an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a write operation provided by an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a merge operation provided by an embodiment of the present invention;

[0047] Figure 4A schematic diagram of the pruning mechanism in a merge operation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0049] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0050] In order to further improve the read and write performance of the log-structured merge tree, the present invention provides a hotspot-aware log-structured merge tree read and write performance optimization method and related equipment. The overall idea is to improve the system's ability to perceive load hotspots, so that hot data is retained in the upper layer of the read and write path as much as possible, shorten the read and write path, reduce the overhead of the internal merge operation of the log-structured merge tree, and improve the read and write performance.

[0051] The following are examples.

[0052] Example 1:

[0053] A hotspot-aware log-structured merge tree read and write performance optimization method, such as Figure 1 As shown, it includes: maintaining a coarse-grained cache and a fine-grained cache in memory, caching the most recently accessed data in the system at the granularity of data blocks and key-value pairs respectively;

[0054] In this embodiment, the coarse-grained cache is a read-only cache that manages data at the granularity of data blocks and can serve range query operations; optionally, the size of the data block is 4KB; the fine-grained cache is a read-write hybrid cache that manages data at the granularity of key-value pairs (of varying sizes) and can serve point query operations and write operations.

[0055] In order to speed up the reading and writing of data in the cache, as a preferred implementation method, in this embodiment, both the coarse-grained cache and the fine-grained cache use partitions combined with hash tables to manage data. Specifically, the cache is divided into multiple partitions, and a hash table is established for each partition, which can effectively improve the concurrent reading and writing capabilities of the cache and further improve the reading and writing performance; when accessing data, first determine the cache partition corresponding to the data, and then use the hash function corresponding to the partition to calculate the hash value to determine the storage location of the data in the partition. The specific partition size and hash function can be set accordingly according to the actual load characteristics. Optionally, in this embodiment, when cache replacement occurs, both the coarse-grained cache and the fine-grained cache use the classic LRU algorithm to determine the replaced data. Accordingly, an LRU queue will be maintained in the cache to quickly determine the eliminated data.

[0056] This embodiment uses fine-grained cache as the first component of the service point query request. Figure 1 As shown, in this embodiment, the point query operation for querying the target key-value pair includes:

[0057] (R1) Query the target key-value pair in the fine-grained cache. If the query is successful, go to step (R4); otherwise, go to step (R2);

[0058] (R2) Access each component in the order of write buffer, read-only write buffer, coarse-grained cache, and log-structured merge tree until the target key-value pair is found. If the target key-value pair is not found in all components, the query operation ends; otherwise, go to step (R3);

[0059] (R3) Cache the queried key-value pair into the fine-grained cache; if the target key-value pair is queried in the log-structured merge tree, cache the data block containing the target key-value pair into the coarse-grained cache;

[0060] (R4) Return the key-value pair found, and the query operation ends;

[0061] This embodiment uses the fine-grained cache as the first component of the service point query request, and maintains a dirty data queue in the fine-grained cache to record the dirty data generated by the update in the fine-grained cache. Figure 2 As shown, in this embodiment, the execution of the write operation includes:

[0062] (W1) Write the key-value pair to be written into the write-before log;

[0063] (W2) Determine whether the key-value pair to be written is located in the fine-grained cache. If so, write the key-value pair to be written into the fine-grained cache by in-place updating, record the data to be written into the dirty data queue, set the corresponding dirty data flag, and then proceed to step (W3); otherwise, write the key-value pair to be written into the write buffer, and then proceed to step (W3);

[0064] (W3) returns the update result and the write operation ends;

[0065] When a data write request hits the fine-grained cache, the dirty data generated will be recorded in the dirty data queue, and the dirty data flag corresponding to the data will also be set. When executing the LRU elimination algorithm, the status of the data to be evicted will be determined based on the dirty data flag corresponding to the data. If the status indicates that it is dirty data, it will be inserted into the write buffer accordingly to ensure that the data can eventually be correctly persisted to the storage device. If the data is clean read-only data, it will be discarded directly to save cache space.

[0066] The above steps (W1) to (W3) complete the foreground processing of the write operation. In addition to steps (W1) to (W3), further processing will be performed by the background thread to ensure that the data is persisted to the storage device. The background processing of the write operation includes:

[0067] After the write buffer is full, the state is changed to read-only state, so that the write buffer is converted into a read-only write buffer. At the same time, a new write buffer is created to continue receiving data for writing;

[0068] The data in the read-only write buffer is flushed back to the persistent storage device in the form of a data table (SSTable) to achieve data storage.

[0069] In this embodiment, the range query operation for searching for key-value pairs within a specified range includes:

[0070] (S1) If the dirty data queue is not empty, all dirty data in the fine-grained cache are read and sorted in batches, and then written into the write buffer in batches, and then the process proceeds to step (S2); otherwise, the process proceeds directly to step (S2);

[0071] Because the fine-grained cache manages data at the granularity of key-value pairs, the data therein is disordered and cannot support range query operations. Before executing the range query operation, this embodiment first writes the dirty data in the fine-grained cache to the write buffer, ensuring that the data queried by the range query operation is the latest data.

[0072] (S2) reading key-value pairs within a specified range from the write buffer and the read-only write buffer, and determining a data block in the daily structure merge tree whose key-value range overlaps with the specified range; if the data block has been cached in the coarse-grained cache, reading the corresponding data block from the coarse-grained cache; otherwise, reading the data block from the log structure merge tree and caching it in the coarse-grained cache; reading key-value pairs within the specified range from the read data block;

[0073] When reading a data block from a log-structured merge tree, this embodiment first determines whether the data block is located in the coarse-grained cache. If so, it is read directly from the coarse-grained cache. This can further shorten the read and write paths with the help of the coarse-grained cache, thereby improving the read and write performance of the log-structured merge tree.

[0074] (S4) merging the read key-value pairs and returning the merged key-value pair set, and the range query operation ends;

[0075] During the merge process, invalid data is discarded and the retained data is sorted.

[0076] Based on the coarse-grained cache and the fine-grained cache, this embodiment implements a dual-grained cache. This embodiment uses the fine-grained cache as the first component to serve read and write requests. It can fully utilize the hot spot characteristics of the load and keep the hot data as much as possible in the upper-level components, thereby effectively shortening the read and write paths, reducing the merge operations on the log structure merge tree, and improving the read and write performance; on this basis, by partitioning the cache and using the memory hash table structure to accelerate the reading and writing of data, the read and write performance can be further improved. The coarse-grained cache and the fine-grained cache work together to provide services for read requests of different granularities. The coarse-grained cache makes up for the disadvantage of the fine-grained cache that cannot efficiently process range queries, and the fine-grained cache makes up for the disadvantage of the low space utilization of the coarse-grained cache. The dual-grained caches work together to adapt to various types of loads.

[0077] To further improve the read and write performance of the log-structured merge tree, this embodiment introduces a hotspot identification mechanism into the merge operation performed in the background. Accordingly, the execution of the merge operation includes:

[0078] Determine the level in the log-structured merge tree where a merge needs to be initiated, calculate the popularity of each data table in that level, and select the data table with the lowest popularity to initiate the merge operation;

[0079] The heat of a data table is the sum of the heats of all data blocks in the data table; the heat of a data block is the sum of the heats of all key-value pairs in the data block;

[0080] In this embodiment, the heat calculation of key-value pairs fully considers the hot spot characteristics of the load. Considering that in the present invention, the data in the fine-grained cache is only inserted into the fine-grained cache during the point query operation, the key-value pairs in the fine-grained cache often have a high read heat. If the key-value pair data has been discarded in the previous merge operation, it means that the data has been written multiple times and thus has a high write heat. Based on this, when performing hot spot identification, this embodiment will identify the data cached by the fine-grained cache in the log structure merge tree as hot read data, and identify the data discarded in the previous merge operation as hot write data. The key-value pair heat calculation method specifically includes:

[0081] Determine whether the key-value pair is in the fine-grained cache. If so, it means that the data is hot read data and its read heat is set to s r Otherwise, its read heat is set to 0; Optionally, in this embodiment, s r =1;

[0082] Determine whether the key-value pair has been discarded in the previous merge operation. If so, it means that the data is hot write data and its write heat is set to s w Otherwise, its write heat is set to 0; Optionally, in this embodiment, s w =1;

[0083] The sum of the read heat and write heat of the key-value pair is taken as the heat of the key-value pair.

[0084] Based on the above operations, this embodiment implements a hot spot identification mechanism in the merge operation. Based on this hot spot identification mechanism, this embodiment selects the data table with the least heat as the initiator of the merge operation, thereby ensuring that the hotter data tables are retained in the upper layer of the log structure merge tree as much as possible to quickly absorb write requests or serve read requests.

[0085] During the execution of the merge operation of the log-structured merge tree, new data blocks are generated, and data blocks already cached in the coarse-grained cache may be invalidated. To improve the cache hit rate, in this embodiment, the execution of the merge operation further includes:

[0086] For new data blocks generated during the merging process, calculate the heat and average heat of each data block, select the data block with heat greater than the average heat, and replace the data block in the coarse-grained cache that has become invalid due to the merging;

[0087] Based on the above operations, this embodiment implements a pre-fetch mechanism for hot data blocks during the merging operation, thereby reducing cache invalidation of the coarse-grained cache caused by the merging operation and improving the cache hit rate.

[0088] The above hotspot identification mechanism and prefetching mechanism can be based on Figure 3The example shown is used to illustrate, Figure 3 In the memory, the heat table is used to record the heat of each data table in the current layer L1 that needs to be merged. In the table, "Table" represents the data table and "Hotness" represents the heat of the corresponding data table. Based on the above hot spot identification mechanism, the coldest data table, namely T3, is selected as the initiator of the merge operation. The merge process will be the data table in the next layer L2 that has an overlapping range with the data table T3, namely Figure 3 T6 and T7 in the table are loaded into memory together and a merge operation is performed. The merge operation generates a new data table, namely Figure 3 In T8 and T9, the original data tables involved in the merge will be deleted lazily, and each data block in each newly generated data table will be calculated during the merge process ( Figure 3 At the same time, some data blocks in the coarse-grained cache will become invalid due to the merge operation. Based on the above prefetch mechanism, the data blocks with a heat greater than the average heat ( Figure 3 B2 and B3 in , pre-fetch it into the coarse-grained cache and replace the data blocks that have failed due to merging.

[0089] This embodiment also designs and implements a pruning mechanism to promptly delete expired and invalid data, avoiding subsequent repeated reading and writing of such invalid data, thereby reducing read-write amplification and optimizing read-write performance. Accordingly, the merge operation also includes:

[0090] For key-value pairs that need to be merged in the log-structured merge tree, if a newer version of the data has been cached in the fine-grained cache, write buffer, or read-only write buffer, the key-value pair will be discarded;

[0091] Based on the above operations, this embodiment implements a pruning mechanism, which can be Figure 4 The example shown is further explained. In the process of merging data, each key-value pair data to be merged is traversed in the order of the keys, and the validity of the key-value pair data is checked accordingly, that is, whether the data is hit in the fine-grained cache, write buffer and read-only write buffer, and whether the version contained in the corresponding fine-grained cache is greater than the version of the data currently being merged. If so, the data is judged to be invalid data, that is, Figure 4 The data 19, 21 and 77 in the merge are directly discarded during the merging process to avoid repeated reading and writing of the data in the future; otherwise, the data is identified as valid data, that is, Figure 4 Data 28 to 56 and 81 cannot be deleted;

[0092] After data pruning is completed, the remaining valid data is written back to the storage device to generate a new data table.

[0093] As a preferred implementation, in this embodiment, the identification of expired data is completed during the process of identifying hot data, that is, during the process of identifying hot and cold data, the validity of the data is checked while determining whether the data is hot data. If the data is found to be invalid, it is promptly deleted from the merging process to avoid subsequent repeated reading and writing of the invalid data. The present invention combines the judgment of data validity with the identification of data hotspots, reducing the additional overhead caused by data heat calculation. At the same time, due to the timely deletion of invalid data, the overhead generated by invalid data in the merging process is greatly reduced, thereby further reducing the read-write amplification and space amplification in the merging process, correspondingly reducing the wear of the storage device and extending the service life of the storage device.

[0094] Example 2:

[0095] A storage controller includes: a computer-readable storage medium and a processor; the computer-readable storage medium stores a computer program; the processor is used to read the computer program in the computer-readable storage medium and execute the hotspot-aware log structure merge tree read and write performance optimization method provided in the above embodiment 1.

[0096] Example 3:

[0097] According to another aspect of the present invention, a storage system based on a log-structured merge tree is provided, comprising: a memory, a storage device, and the storage controller provided in the above-mentioned embodiment 2.

[0098] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A hotspot-aware log-structured merge tree read and write performance optimization method, characterized in that: include: Maintain a coarse-grained cache and a fine-grained cache in memory, caching the most recently accessed data in the system at the granularity of data blocks and key-value pairs respectively; Furthermore, the point query operation for querying the target key-value pair includes: (R1) Querying the target key-value pair in the fine-grained cache, if the query is successful, proceeding to step (R4); otherwise, proceeding to step (R2); (R2) sequentially accessing each component in the order of the write buffer, the read-only write buffer, the coarse-grained cache, and the log-structured merge tree until the target key-value pair is found. If the target key-value pair is not found in any component, the query operation ends; otherwise, proceeding to step (R3); (R3) caching the queried key-value pair in the fine-grained cache; if the target key-value pair is found in the log-structured merge tree, caching the data block containing the target key-value pair in the coarse-grained cache; (R4) Return the key-value pair found, and the query operation ends; The fine-grained cache also maintains a dirty data queue for recording dirty data generated by updates in the fine-grained cache; Furthermore, the execution of the write operation includes: (W1) Write the key-value pair to be written into the write-ahead log; (W2) Determine whether the key-value pair to be written is located in the fine-grained cache. If so, write the key-value pair to be written into the fine-grained cache by in-place updating, record the data to be written into the dirty data queue, and set the corresponding dirty data flag, and then proceed to step (W3); otherwise, write the key-value pair to be written into the write buffer, and then proceed to step (W3); (W3) Returns the update result and the write operation ends.

2. The hotspot-aware log-structured merge tree read-write performance optimization method according to claim 1, characterized in that: Range query operation is used to query key-value pairs within a specified range. Its execution includes: (S1) If the dirty data queue is not empty, all dirty data in the fine-grained cache are read and sorted in batches, and then written into the write buffer in batches, and then the process proceeds to step (S2); otherwise, the process proceeds directly to step (S2); (S2) reading key-value pairs within the specified range from the write buffer and the read-only write buffer, and determining a data block in the log-structured merge tree whose key-value range overlaps with the specified range; if the data block has been cached in the coarse-grained cache, reading the corresponding data block from the coarse-grained cache; otherwise, reading the data block from the log-structured merge tree and caching it in the coarse-grained cache; and reading key-value pairs within the specified range from the read data block; (S4) The key-value pairs read are merged and the merged key-value pair set is returned, and the range query operation ends.

3. The hotspot-aware log-structured merge tree read-write performance optimization method according to claim 2, characterized in that: The coarse-grained cache is divided into a plurality of first partitions, each of which uses a hash table to manage data blocks; the fine-grained cache is divided into a plurality of second partitions, each of which uses a hash table to manage key-value pairs.

4. The hotspot-aware log-structured merge tree read-write performance optimization method according to claim 2 or 3, characterized in that: The merging operation of the log structure merge tree includes: Determine the level in the log structure merge tree where a merge needs to be initiated, calculate the popularity of each data table in the level, and select the data table with the smallest popularity to initiate a merge operation; The heat of a data table is the sum of the heats of all data blocks in the data table; the heat of a data block is the sum of the heats of all key-value pairs in the data block.

5. The hotspot-aware log-structured merge tree read-write performance optimization method according to claim 4, characterized in that: In the log structure merge tree, the heat calculation method of the key-value pair includes: Determine whether the key-value pair is in the fine-grained cache. If so, set its read heat to s r , otherwise, set its read heat to 0; s r >0; Determine whether the key-value pair has been discarded in the previous merge operation. If so, set its write heat to s w , otherwise, set its write heat to 0; s w > 0; The sum of the read heat and write heat of the key-value pair is taken as the heat of the key-value pair.

6. The hotspot-aware log-structured merge tree read-write performance optimization method according to claim 4, characterized in that: The merging operation of the log structure merge tree further includes: For new data blocks generated during the merging process, the heat and average heat of each data block are calculated, and data blocks with heat greater than the average heat are selected to replace the data blocks in the coarse-grained cache that have become invalid due to the merging.

7. The hotspot-aware log-structured merge tree read-write performance optimization method according to claim 4, characterized in that: The merging operation of the log structure merge tree further includes: For the key-value pairs that need to be merged in the log-structured merge tree, if updated versions of the data have been cached in the fine-grained cache, write buffer, or read-only write buffer, the key-value pairs are discarded.

8. A storage controller, characterized in that: include: Computer-readable storage media and processor; The computer-readable storage medium stores a computer program; the processor is used to read the computer program in the computer-readable storage medium and execute the hotspot-aware log-structured merge tree read and write performance optimization method described in any one of claims 1 to 7.

9. A storage system based on a log-structured merge tree, characterized in that: include: A memory, a storage device, and the storage controller according to claim 8.

Images