A key-value storage method and system based on SSD-SMR hybrid storage

Abstract

The application discloses a kind of based on SSD-SMR hybrid storage's key value storage method and system, (Key, Real_Value) is written into the corresponding Zone in HM-SMR, then (Key, Virtual_Value) is written into MemTable, subsequent again deposit into the SSTable file of SSD, complete write operation;Search Key corresponding Virtual_Value in memory and LSM-tree, analyze Virtual_Value, according to the analysis result, obtain the real value from Zone, complete read operation;Invalid SSTable file in LSM-tree is removed by merging operation, and garbage collection mechanism is designed for HM-SMR, when there is Zone to meet the recovery condition, remove the invalid data in the Zone.The application alleviates the write amplification problem generated by the combination of LSM-tree type key value storage technology and HM-SMR disk, reduces the cost of system, improves the stability and overall performance of system.

Description

Technical Field

[0001] This invention belongs to the field of computer system architecture technology, specifically relating to a key-value storage method and system based on SSD-SMR hybrid storage. Background Technology

[0002] Log-Structured Merge Tree (LSM-tree) based key-value storage technology is currently the most widely used NoSQL database technology. Compared to key-value storage technologies using B-trees or B+ trees, LSM-tree key-value storage transforms random writes into batch sequential writes through merging operations, reducing data updates to disk. Therefore, LSM-tree based key-value storage technology offers superior random write performance, a feature that makes it crucial in modern massive data management.

[0003] To build cost-effective, high-capacity storage systems, combining LSM-tree key-value storage technology with low-cost, high-capacity storage devices is an effective solution. Among the many technologies for increasing disk capacity, watt-hour recording technology can significantly improve storage density per unit area while minimizing changes to the disk structure, making it one of the best choices for building high-capacity key-value storage systems.

[0004] Shingled Magnetic Recording (SMR), also known as tile recording, is a promising data storage technology. Unlike traditional disk storage technologies, SMR does not maintain a guard distance between adjacent tracks; instead, it overlaps the tracks to increase the disk's areal density. Within the same physical size, SMR can significantly expand the disk's storage space, meeting the massive data storage needs of data centers.

[0005] For different application scenarios, the industry classifies current mainstream SMR disks into three types: DM-SMR, HM-SMR, and HA-SMR. HM-SMR specifies that zones on the disk can only accept sequential writes, maximizing the disk bandwidth of SMR. Furthermore, HM-SMR displays parameters such as zone size, zone offset, and write pointer position to upper-layer applications, which helps the host build a storage system tailored to its characteristics, thereby achieving stable and predictable performance. The characteristics of HM-SMR are highly compatible with LSM-trees, making it an ideal choice for expanding the capacity of LSM-tree key-value storage systems. Therefore, this patent selects HM-SMR disks as one of the basic storage devices in its optimization scheme.

[0006] Combining LSM-tree key-value storage technology with HM-SMR disks is beneficial for building low-cost, high-capacity key-value storage systems, but both technologies suffer from write amplification. LSM-trees require continuous merge operations to consolidate data and re-store the merged data, thus cleaning up expired data and ensuring read performance. However, frequent merge operations generate significant additional write overhead, exacerbating write amplification and reducing system performance. Meanwhile, SMR disks, limited by their overlapping track structure, must read data from all downstream tracks of the track to be modified before updating any data, and then write the data back to the original track in sequence. This "read-modify-write" operation causes severe write amplification, leading to overall system performance degradation. Therefore, simply combining LSM-tree key-value storage technology with HM-SMR further exacerbates write amplification, causing unpredictable performance fluctuations during system operation.

[0007] Google's LevelDB is one of the most representative LSM-tree key-value stores, and its persistent key-value storage solution based on BigTable can efficiently handle hundreds of millions of data points. However, the LSM-tree, composed of SSTable (Sorted String Table) files, continuously performs merging operations between adjacent levels in the background, generating significant additional write overhead. Deploying LevelDB directly on HM-SMR disks further exacerbates write amplification, especially when writing key-value pairs with values ​​as small as 4KB. The write amplification becomes increasingly severe as the amount of data written increases. When writing 40GB of data, the write amplification factor reaches 13.63, which seriously impacts system performance. Therefore, to build a stable storage system using LSM-tree key-value storage technology and HM-SMR disks, it is essential to minimize write amplification as much as possible. Summary of the Invention

[0008] The technical problem to be solved by the present invention is to provide a key-value storage method and system based on SSD-SMR hybrid storage, which addresses the shortcomings of the prior art and solves the write amplification problem caused by the combination of LSM-tree key-value storage technology and HM-SMR disk, thereby reducing system cost and improving system stability and overall performance.

[0009] The present invention adopts the following technical solution:

[0010] A key-value storage method based on SSD-SMR hybrid storage includes the following steps:

[0011] S1. Write (Key, Real_Value) to the corresponding Zone in HM-SMR, then write (Key, Virtual_Value) to MemTable, and then store it in the SSTable file of SSD to complete the write operation;

[0012] S2. Search for the Virtual_Value corresponding to the Key in memory and LSM-tree, analyze the Virtual_Value, and retrieve the actual value from the Zone based on the parsing results to complete the read operation;

[0013] S3. Remove invalid SSTable files in the LSM-tree through a merge operation, and design a garbage collection mechanism for HM-SMR. When a Zone meets the collection conditions, remove the invalid data in that Zone.

[0014] Specifically, step S1 is as follows:

[0015] When writing a key-value pair (Key, Value), HMDB determines the group Key-range i to which it belongs based on the Key, and then writes (Key, Real_Value) to the Current-Zone of Key-range i;

[0016] If the Current-Zone is full or its remaining space is insufficient, HMDB allocates a new Zone for Key-range i to serve as the Current-Zone before writing data.

[0017] After a successful write, HMDB generates an index Virtual_Value based on the key-value pair size (length), the Zone number (num), and the offset (pos), and writes (Key, Real_Value) to MemTable; when the amount of data in MemTable reaches a threshold, it is converted into an Immutable MemTable.

[0018] Then HMDB generates an SSTable file based on the data in the Immutable MemTable, writes it to level 0 of the LSM-tree, and waits for subsequent merge operations. At this point, all key-value pair (Key, Value) write operations are completed.

[0019] Furthermore, in the process of deleting key-value pairs (Key, Value), Real_Value is replaced with the deletion marker.

[0020] Specifically, step S2 is as follows:

[0021] When reading the value corresponding to a key, HMDB searches in ascending order of data age. First, it searches the MemTable; if the target is found, it proceeds to the next stage; otherwise, it continues searching the ImmutableMemTable. If neither is found, HMDB searches the LSM-tree from lowest to highest level. If the target is not found in the LSM-tree (key, value), and the system does not have a key-value pair, after finding the Virtual_Value corresponding to the key, HMDB parses out the length, num, and pos, then uses these values ​​to locate the corresponding Zone, reads the Real_Value corresponding to the key, and completes the read operation.

[0022] Specifically, step S3 is as follows:

[0023] S301 and HMDB set two parameters for each Zone: Total_volume (total amount of data written) and Invalid_volume (total amount of invalid data). Each time a key-value pair of size length is written to a Zone, Total_volume is incremented by length until the Zone stops receiving data.

[0024] S302. When performing a garbage collection operation on a Zone, HMDB reads key-value pairs one by one from the beginning of the Zone and determines whether they are valid.

[0025] Furthermore, in step S301, if frequent garbage collection is performed during the update of Invalid_volume, it will put a considerable burden on system performance. Therefore, HMDB takes into account the hotness of data and prioritizes the garbage collection operation for Zones storing hot data. When the LSM-tree encounters key and value indexes to be cleared during merging, it parses out the Zone number num and key-value pair size length, and then updates the Invalid_volume value of that Zone. If the Invalid_volume of a Zone is greater than or equal to the Total_volume, and the Zone is not a Current Zone, then garbage collection can be performed on it.

[0026] Furthermore, the calculation of data popularity is as follows:

[0027] Heat Key-Value =W read *Frequency read +W update *Frequency update

[0028] Among them, Heat Key-Value W represents the popularity of key-value pairs (Key, Value). read and W update These represent the weights of read and update operations, respectively, and the frequency. read and Frequency update These represent the frequency of reading and updating key-value pairs (Key, Value), respectively.

[0029] The increment is calculated as follows:

[0030] Increment Key-Value =length * Heat Key-Value

[0031] Among them, Increment Key-Value This represents the increment of the `Invalid_volume` of the zone containing the key-value pair (Key, Value) after it has been invalidated. `length` is the actual size of the key-value pair. `Heat` Key-Value This represents the popularity of key-value pairs.

[0032] Furthermore, in step S302, the validity determination process for key-value pairs (Key, Value) is as follows:

[0033] First, check if Real_Value is marked for deletion. If not, search in memory and LSM-tree based on the Key. When the corresponding value index Virtual_Value is found and the number num parsed from the index matches the current Zone number, the key-value pair is valid data. HMDB rewrites it to the system according to the write operation rules. Otherwise, the key-value pair has expired.

[0034] Once all key-value pairs in a Zone have been read, the Zone is released as an idle Zone for later use, thus completing the garbage collection operation for that Zone.

[0035] Specifically, the ratio of SSD solid-state drives to HM-SMR disks is calculated as follows:

[0036]

[0037] Capacity SSD =Ratio Key-Value *Capacity HM-SMR

[0038] Among them, Ratio Key-Value Size represents the ratio of storage space used by the SSD to that used by the HM-SMR when storing key-value pairs (Key, Value).Key Size represents the size of the key. Virtual_Value Size represents the size of the value index. Real_Value Capacity represents the actual value. SSD and Capacity HM-SMR These represent the SSD and HM_SMR space sizes used for storing (Key, Value), respectively.

[0039] Secondly, embodiments of the present invention provide a key-value storage system based on SSD-SMR hybrid storage, comprising:

[0040] The write module writes (Key, Real_Value) to the corresponding Zone in HM-SMR, then writes (Key, Virtual_Value) to MemTable, and finally stores it in the SSTable file of the SSD to complete the write operation;

[0041] The read module searches for the Virtual_Value corresponding to the Key in memory and the LSM-tree, analyzes the Virtual_Value, and retrieves the actual value from the Zone based on the parsing results to complete the read operation;

[0042] The recycling module removes invalid SSTable files from the LSM-tree through a merge operation and designs a garbage collection mechanism for HM-SMR, which removes invalid data from a Zone when the zone meets the recycling criteria.

[0043] Compared with the prior art, the present invention has at least the following beneficial effects:

[0044] A key-value storage method based on SSD-SMR hybrid storage aims to reduce system write amplification. It employs the core concept of key-value separation, dividing key-value pairs (Key, Value) into two parts: the actual value and the index. During write operations, the actual value pair (Key, Real_Value) is written to the HM-SMR, while the index pair (Key, Virtual_Value) is written to memory and the SSD, reducing the additional write overhead caused by subsequent LSM-tree merging operations. During read operations, the target is quickly located by searching memory and the LSM-tree. During garbage collection, invalid data in the SSD is removed using merging operations, and the invalid data ratio calculation method based on hot / cold data identification determines the partitions in the HM-SMR that need to be reclaimed, improving the device's space utilization.

[0045] Furthermore, since the size of Real_Value is generally much larger than the size of Virtual_Value, writing (Key, Real_Value) to HM-SMR can effectively take advantage of the device's low cost and large capacity; while writing (Key, Virtual_Value) to memory and LSM-tree can improve merging efficiency and help reduce the system's write amplification factor.

[0046] Furthermore, when deleting key-value pairs (Key, Value), to avoid the query overhead caused by direct deletion, Real_Value is replaced with a deletion flag, and write operations continue. Then, the actual deletion is completed through subsequent merge and garbage collection operations.

[0047] Furthermore, the target key-value pairs are searched in the order of MemTable, Immutable MemTable, and LSM-tree to ensure that the retrieved data is the latest version.

[0048] Furthermore, LSM-tree automatically deletes invalid (Key, Virtual_Value) records during the merge operation, while the corresponding (Key, Real_Value) records in HM-SMR require additional garbage collection operations to reclaim space. Therefore, the total amount of data written (Total_volume) and the total amount of invalid data (Invalid_volume) are set as the criteria for garbage collection; at the same time, the validity judgment is used to determine which operation to perform on the data during collection.

[0049] Furthermore, to avoid frequent garbage collection from burdening the overall system performance, key-value pairs are identified as hot or cold. The Invalid_volume value of a partition is updated based on the hotness or coldness of the data. When the Invalid_volume value of a partition is greater than or equal to the Total_volume value, and the partition is no longer being written to, garbage collection is performed on it.

[0050] Furthermore, for identifying hot and cold key-value data, the impact of update frequency and query frequency are taken into account, and data popularity is used as one of the criteria for garbage collection operations. The increment of the total amount of invalid data is set as the product of the size of a single invalid data and its popularity.

[0051] Furthermore, to ensure the correct operation of garbage collection, it is necessary to perform a validity check on the (Key, Real_Value) in the partition, then write back valid data and discard invalid data.

[0052] Furthermore, the ratio of SSD to HM-SMR disk depends on the size comparison of the key and value in the key-value pair. The greater the difference between the key and value, the less SSD space is required for the same HM-SMR disk space, and the lower the system's write amplification factor.

[0053] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0054] In summary, this invention adopts the idea of ​​key-value separation, writes the real value of the key-value pair into HM-SMR, and writes its index into memory and SSD. Garbage collection and merging operations based on hot and cold identification are used to reclaim these two types of data, thereby improving the merging efficiency of LSM-tree, reducing the write amplification factor, and improving the overall performance of the system.

[0055] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0056] Figure 1 This is a diagram of the overall architecture of the present invention;

[0057] Figure 2 Write an operation flowchart for this invention;

[0058] Figure 3 This is a flowchart of the read operation of the present invention;

[0059] Figure 4 This is a flowchart of the waste recycling process of the present invention. Detailed Implementation

[0060] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0061] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0062] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0063] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0064] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0065] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0066] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0067] This invention provides a key-value storage method based on SSD-SMR hybrid storage, and a data management scheme (Host-Managed Database, HMDB) with key-value separation as the core strategy. By separating and managing keys and values, it alleviates the write amplification problem caused by merging operations in LSM-tree, thereby building a low-cost, high-performance, large-capacity key-value storage system.

[0068] Please see Figure 1 Generate two new key-value pairs based on the given key-value pairs (Key, Value):

[0069] (Key, Real_Value) and (Key, Virtual_Value).

[0070] Real_Value is the same as Value, representing the actual value;

[0071] Virtual_Value is an index of Real_Value, used to determine the location of the real value on the HM-SMR disk.

[0072] In memory, MemTable is used to store (Key, Virtual_Value), enabling fast read and write operations; ImmutableMemTable is responsible for persisting data that has reached its limit to an SSTable file.

[0073] Regarding storage devices, because the LSM-tree composed of SSTable files requires frequent merging operations, HMDB stores it on SSDs and stores the (Key, Real_Value) data that does not require frequent updates on HM-SMR disks. This storage method leverages the high random read / write speed of SSDs to increase index lookup speed, while reducing the adverse impact of data updates on HM-SMR disks and mitigating write amplification.

[0074] In key-value pairs (Key, Value), the size of the Key is smaller than the size of the Value, and the size of the Virtual_Value in HMDB is relatively small compared to the actual Value. Therefore, the SSD storage space used in HMDB is much smaller than that in HM-SMR. In HMDB, the ratio of SSD solid-state drives to HM-SMR disks is calculated using formulas (1) and (2).

[0075]

[0076] Capacity SSD =Ratio Key-Value *Capacity HM-SMR (2)

[0077] Among them, Ratio Key-Value Vize represents the ratio of storage space used by the SSD to that used by the HM-SMR when storing key-value pairs (Key, Value). Key Size represents the size of the key. Virtual_Value Size represents the size of the value index.Real_Value Capacity represents the actual value. SSD and Capacity HM-SMR Ratio represents the SSD and HM_SMR space sizes used for storing (Key, Value), respectively. As the amount of data in the Value increases, Ratio... Key-Value It also gets smaller and smaller. As the value increases from 1KB to 16KB, the ratio... Key-Value The value will decrease from 0.035 to 0.002.

[0078] In summary, HMDB requires only a small amount of SSD space to build a cost-effective, high-capacity key-value storage system.

[0079] HMDB divides the zones in an HM-SMR disk into m groups based on key-value ranges. Each zone only receives key-value pairs within a specific range, facilitating data management. Furthermore, HMDB stipulates that only one zone in each group can receive data at any given time; this zone is called the Current Zone. Because traditional file systems are incompatible with the HM-SMR partition structure, HMDB uses the Libzbc user library provided by HGST to access HM-SMR. The following sections will describe HMDB's write operation implementation, read operation implementation, and garbage collection implementation.

[0080] This invention discloses a key-value storage method based on SSD-SMR hybrid storage, comprising the following steps:

[0081] S1, Write Operation

[0082] HMDB write operations are divided into two steps: the first step is to write (Key, Real_Value) to the corresponding Zone in HM-SMR, and the second step is to write (Key, Virtual_Value) to MemTable, which is then stored in the SSTable file of the SSD.

[0083] Please see Figure 2 The specific steps are as follows:

[0084] When writing a key-value pair (Key, Value), HMDB first determines the group Key-range i to which the key belongs, and then writes (Key, Real_Value) to the Current-Zone of Key-range i.

[0085] If the Current-Zone is full or its remaining space is insufficient, HMDB allocates a new Zone for Key-range i to serve as the Current-Zone before writing data.

[0086] After a successful write, HMDB generates an index Virtual_Value based on the key-value pair size (length), the Zone number (num), and the offset (pos), and writes (Key, Virtual_Value) to MemTable; when the amount of data in MemTable reaches a threshold, it is converted into an Immutable MemTable.

[0087] Then HMDB generates an SSTable file based on the data in the Immutable MemTable, writes it to level 0 of the LSM-tree, and waits for subsequent merge operations. At this point, all key-value pair (Key, Value) write operations are completed.

[0088] The process of deleting key-value pairs (Key, Value) is basically the same as described above, except that Real_Value is replaced with a deletion marker. Therefore, HMDB's deletion operation does not actually delete the data; it only informs the user that the key-value pair no longer exists at the logical level. The physical deletion is accomplished through subsequent merging and garbage collection.

[0089] S2, Read Operation

[0090] HMDB read operations are also divided into two stages. The first stage involves searching for the Virtual_Value corresponding to the Key in memory and the LSM-tree. The second stage involves analyzing the Virtual_Value and retrieving the actual value from the Zone based on the parsing results.

[0091] Please see Figure 3 The specific steps are as follows:

[0092] When reading the value corresponding to a key, HMDB searches in ascending order of data age. That is, it first searches the MemTable.

[0093] If the target is found, proceed to the next stage of the operation; otherwise, continue searching for Immutable MemTable.

[0094] If neither part is matched, HMDB searches the LSM-tree layer by layer from low to high.

[0095] If the target cannot be found in the LSM-tree, it means that the key-value pair (Key, Value) does not exist in the system. After finding the Virtual_Value corresponding to the Key, HMDB parses out the length, num, and pos from it, and then locates the corresponding Zone based on these three parameters, reads the Real_Value corresponding to the Key, and completes the read operation.

[0096] S3, Waste Recycling

[0097] As the storage system continues to operate, some key-value pairs will be deleted or updated, becoming invalid and expired data. Therefore, the system needs to perform garbage collection in a timely manner. In SSDs, LSM-trees clear invalid data from SSTable files through merge operations, which can serve as a garbage collection mechanism. However, key-value pairs stored in HM-SMR cannot be reclaimed in this way. Therefore, to alleviate the problem of wasted space, HMDB designed a garbage collection mechanism specifically for HM-SMR. When a Zone meets the reclamation criteria, HMDB will perform garbage collection on it.

[0098] Please see Figure 4 The specific steps are as follows:

[0099] S301, Zone Recycling Conditions

[0100] HMDB sets two parameters for each Zone: Total_volume (total amount of data written) and Invalid_volume (total amount of invalid data). Each time a key-value pair of size length is written to a Zone, Total_volume is incremented by length until that Zone stops receiving data.

[0101] Updating Invalid_volume is more complex. Since garbage collection updates the entire Zone, frequent collection can place a significant burden on system performance. Therefore, HMDB considers the hotness of data and prioritizes collecting data from Zones storing frequently accessed data. The calculation of data hotness is shown in formula (3).

[0102] Heat Key-Value =W read *Frequency read +W update *Frequency update (3)

[0103] Among them, Heat Key-Value W represents the popularity of key-value pairs (Key, Value). read and W updateThese represent the weights of read and update operations, respectively, and the frequency. read and Frequency update These represent the read frequency and update frequency of key-value pairs (Key, Value), respectively.

[0104] When merging, if an LSM-tree encounters a key-value index to be cleared, it parses the Zone number (num) and key-value pair size (length) from it, and then updates the Invalid_volume value of that Zone. The increment is calculated as shown in formula (4).

[0105] Increment Key-Value =length * Heat Key-Value (4)

[0106] Among them, Increment Key-Value This represents the increment of the `Invalid_volume` of the zone containing the key-value pair (Key, Value) after it has been invalidated. `length` is the actual size of the key-value pair. `Heat` Key-Valuw This represents the popularity of key-value pairs.

[0107] If a Zone's Invalid_volume is greater than or equal to Total_volume, and that Zone is not a Current-Zone, then garbage collection can be performed on it. Furthermore, if the data is too hot, a Zone might be garbage collected even when there are very few invalid key-value pairs. Therefore, HMDB is for Heat. Key-Value A maximum heat limit, max_heat, is set to avoid unnecessary garbage collection operations.

[0108] S302, Zone recycling process.

[0109] When performing garbage collection on a Zone, HMDB reads key-value pairs one by one from the beginning of the Zone and determines whether they are valid.

[0110] The validity process for key-value pairs (Key, Value) is as follows:

[0111] First, check if Real_Value is marked for deletion. If it is, the key-value pair has been deleted and no action is taken. Otherwise, search in memory and the LSM-tree based on the key. If a corresponding value index Virtual_Value is found, and the parsed number num matches the current Zone number, the key-value pair is valid, and HMDB will rewrite it to the system according to write operation rules. Otherwise, the key-value pair has expired. Once all key-value pairs in this Zone have been read, it is released, making the Zone free for later use. This completes the garbage collection operation for a Zone.

[0112] In another embodiment of the present invention, a key-value storage system based on SSD-SMR hybrid storage is provided. This system can be used to implement the above-mentioned key-value storage method based on SSD-SMR hybrid storage. Specifically, the key-value storage system based on SSD-SMR hybrid storage includes a write module, a read module, and a recycling module.

[0113] The write module writes (Key, Real_Value) to the corresponding Zone in HM-SMR, then writes (Key, Virtual_Value) to MemTable, and finally stores it in the SSTable file of the SSD to complete the write operation.

[0114] The read module searches for the Virtual_Value corresponding to the Key in memory and the LSM-tree, analyzes the Virtual_Value, and retrieves the actual value from the Zone based on the parsing results to complete the read operation;

[0115] The recycling module removes invalid SSTable files from the LSM-tree through a merge operation and designs a garbage collection mechanism for HM-SMR, which removes invalid data from a Zone when the zone meets the recycling criteria.

[0116] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, the computer program including program instructions, and the processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a key-value storage method based on SSD-SMR hybrid storage, including:

[0117] The (Key, Real_Value) is written to the corresponding Zone in HM-SMR, and then (Key, Virtual_Value) is written to MemTable, and subsequently stored in the SSTable file of the SSD, completing the write operation. The Virtual_Value corresponding to the Key is searched in memory and LSM-tree, the Virtual_Value is analyzed, and the real value is obtained from the Zone based on the parsing results, completing the read operation. Invalid SSTable files in LSM-tree are cleared through a merge operation, and a garbage collection mechanism is designed for HM-SMR. When a Zone meets the reclamation conditions, invalid data in that Zone is cleared.

[0118] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0119] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the key-value storage method based on SSD-SMR hybrid storage in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor to perform the following steps:

[0120] The (Key, Real_Value) is written to the corresponding Zone in HM-SMR, and then (Key, Virtual_Value) is written to MemTable, and subsequently stored in the SSTable file of the SSD, completing the write operation. The Virtual_Value corresponding to the Key is searched in memory and LSM-tree, the Virtual_Value is analyzed, and the real value is obtained from the Zone based on the parsing results, completing the read operation. Invalid SSTable files in LSM-tree are cleared through a merge operation, and a garbage collection mechanism is designed for HM-SMR. When a Zone meets the reclamation conditions, invalid data in that Zone is cleared.

[0121] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0122] The table below shows the results of random write experiments conducted by this invention in scenarios with different Value values.

[0123]

[0124] When the size of the written value is the same, the random write performance of this invention is significantly higher than that of LevelDB; and as the value size increases, the improvement in random write performance becomes more significant.

[0125] In summary, this invention provides a key-value storage method and system based on SSD-SMR hybrid storage, which reduces the amount of extra write data generated during LSM-tree merging operations, lowers system latency, and improves read and write performance to some extent. Simultaneously, this invention reduces the number of "read-modify-write" operations on the HM-SMR disk, reducing additional write overhead while effectively utilizing its capacity. In practical applications, this invention can alleviate the write amplification problem caused by the combination of LSM-tree key-value storage technology and HM-SMR disks, reduce system costs, and improve system stability and overall performance.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0127] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0129] In the embodiments provided by this invention, it should be understood that the disclosed devices / terminals and methods can be implemented in other ways. For example, the device / terminal embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0132] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0133] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0134] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0135] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0136] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A key-value storage method based on SSD-SMR hybrid storage, characterized in that, Includes the following steps: S1. Write (Key, Real_Value) to the corresponding Zone in HM-SMR, then write (Key, Virtual_Value) to MemTable, and finally store it in the SSD's SSTable file to complete the write operation. Specifically: When writing a key-value pair (Key, Value), HMDB determines the group Key-range i to which it belongs based on the Key, and then writes (Key, Real_Value) to the Current-Zone of Key-range i; If the Current-Zone is full or its remaining space is insufficient, HMDB allocates a new Zone for Key-range i to serve as the Current-Zone before writing data. After a successful write, HMDB generates an index Virtual_Value based on the key-value pair size (length), the Zone number (num), and the offset (pos), and writes (Key, Real_Value) to MemTable; when the amount of data in MemTable reaches a threshold, it is converted into an Immutable MemTable. Then HMDB generates an SSTable file based on the data in the Immutable MemTable, writes it to level 0 of the LSM-tree, and waits for subsequent merge operations. At this point, all key-value pair (Key, Value) write operations are completed. S2. Search for the Virtual_Value corresponding to the Key in memory and the LSM-tree, analyze the Virtual_Value, and retrieve the actual value from the Zone based on the parsing results to complete the read operation. Specifically: When reading the value corresponding to a key, HMDB searches in ascending order of data age; that is, it first searches the MemTable; if the target is found, it proceeds to the next stage of operation; otherwise, it continues searching the ImmutableMemTable; if neither is found, HMDB searches the LSM-tree layer by layer from low to high; if the target is not found in the LSM-tree, the key-value pair (Key, Value) does not exist in the system, and after finding the Virtual_Value corresponding to the key, HMDB parses out the length, num, and pos, and then locates the corresponding Zone based on the length, num, and pos, reads the Real_Value corresponding to the key, and completes the read operation; S3. Invalid SSTable files in the LSM-tree are cleared through merge operations, and a garbage collection mechanism is designed for HM-SMR. When a Zone meets the collection criteria, invalid data in that Zone is cleared. Specifically: S301 and HMDB set two parameters for each Zone: Total_volume (total amount of data written) and Invalid_volume (total amount of invalid data). Each time a key-value pair of size length is written to a Zone, Total_volume is incremented by length until the Zone stops receiving data. S302. When performing a garbage collection operation on a Zone, HMDB reads key-value pairs one by one from the beginning of the Zone and determines whether they are valid.

2. The key-value storage method based on SSD-SMR hybrid storage according to claim 1, characterized in that, In the process of deleting key-value pairs (Key, Value), Real_Value is replaced with the deletion marker.

3. The key-value storage method based on SSD-SMR hybrid storage according to claim 1, characterized in that, In step S301, during the update of Invalid_volume, priority is given to recycling the Zones storing hot data. When the LSM-tree encounters key and value indices to be cleared during merging, it parses out the Zone number num and key-value pair size length, and then updates the Invalid_volume value of that Zone. If the Invalid_volume of a Zone is greater than or equal to Total_volume, and the Zone is not a Current-Zone, then garbage collection can be performed on it.

4. The key-value storage method based on SSD-SMR hybrid storage according to claim 3, characterized in that, The data popularity is calculated as follows: in, Represents the popularity of key-value pairs (Key, Value). and These represent the weights of the read and update operations, respectively. and These represent the frequency of reading and updating key-value pairs (Key, Value), respectively. The increment is calculated as follows: in, This represents the increment of the Invalid_volume of the Zone after the key-value pair (Key, Value) is invalidated. The actual size of the key-value pairs. This represents the popularity of key-value pairs.

5. The key-value storage method based on SSD-SMR hybrid storage according to claim 1, characterized in that, In step S302, the validity determination process for key-value pairs (Key, Value) is as follows: First, check if Real_Value is marked for deletion. If not, search in memory and LSM-tree based on the Key. When the corresponding value index Virtual_Value is found and the number num parsed from the index matches the current Zone number, the key-value pair is valid data. HMDB rewrites it to the system according to the write operation rules. Otherwise, the key-value pair has expired. Once all key-value pairs in a Zone have been read, the Zone is released as a free Zone for later use, thus completing the garbage collection operation for that Zone.

6. The key-value storage method based on SSD-SMR hybrid storage according to any one of claims 1 to 5, characterized in that, The ratio of SSD solid-state drives to HM-SMR disks is calculated as follows: in, This represents the ratio of storage space used by the SSD to that used by the HM-SMR when storing key-value pairs (Key, Value). Represents the size of the bond. Represents the size of the value index. Represents the magnitude of the true value. and These represent the SSD and HM_SMR space sizes used for storing (Key, Value), respectively.

7. A key-value storage system based on SSD-SMR hybrid storage, characterized in that, include: The write module writes (Key, Real_Value) to the corresponding Zone in HM-SMR, then writes (Key, Virtual_Value) to MemTable, and finally stores it in the SSD's SSTable file, completing the write operation. Specifically: When writing a key-value pair (Key, Value), HMDB determines the group Key-range i to which it belongs based on the Key, and then writes (Key, Real_Value) to the Current-Zone of Key-range i; If the Current-Zone is full or its remaining space is insufficient, HMDB allocates a new Zone for Key-range i to serve as the Current-Zone before writing data. After a successful write, HMDB generates an index Virtual_Value based on the key-value pair size (length), the Zone number (num), and the offset (pos), and writes (Key, Real_Value) to MemTable; when the amount of data in MemTable reaches a threshold, it is converted into an Immutable MemTable. Then HMDB generates an SSTable file based on the data in the Immutable MemTable, writes it to level 0 of the LSM-tree, and waits for subsequent merge operations. At this point, all key-value pair (Key, Value) write operations are completed. The read module searches for the Virtual_Value corresponding to the Key in memory and the LSM-tree, analyzes the Virtual_Value, retrieves the actual value from the Zone based on the parsing results, and completes the read operation. Specifically: When reading the value corresponding to a key, HMDB searches in ascending order of data age; that is, it first searches the MemTable; if the target is found, it proceeds to the next stage of operation; otherwise, it continues searching the ImmutableMemTable; if neither is found, HMDB searches the LSM-tree layer by layer from low to high; if the target is not found in the LSM-tree, the key-value pair (Key, Value) does not exist in the system, and after finding the Virtual_Value corresponding to the key, HMDB parses out the length, num, and pos, and then locates the corresponding Zone based on the length, num, and pos, reads the Real_Value corresponding to the key, and completes the read operation; The garbage collection module removes invalid SSTable files from the LSM-tree through merge operations and designs a garbage collection mechanism for HM-SMR. When a Zone meets the collection criteria, invalid data in that Zone is removed. Specifically: HMDB sets two parameters for each Zone: Total_volume (total amount of data written) and Invalid_volume (total amount of invalid data). Each time a key-value pair of size length is written to a Zone, Total_volume is incremented by length until the Zone stops receiving data. When performing garbage collection on a Zone, HMDB reads key-value pairs one by one from the beginning of the Zone and determines whether they are valid.

Images