A data storage method and device based on address redirection

Abstract

The present invention discloses a data storage method and device based on address redirection. The method comprises receiving a write request, extracting a data block identifier, data offset, and data length from the write request, determining whether the write is a large block or a small block based on the data length, querying a metadata database based on the data block identifier and data offset, and determining whether the write is a new write operation or an overwrite write operation; obtaining the bandwidth, IOPS, and latency of the HDD device within the current sliding time window to determine the load of the HDD device; writing data based on the load of the HDD device; and persisting data distribution information and address space usage information to the metadata database to complete data storage. The present invention utilizes the characteristics of the disk and the current load to perform IO scheduling between the SSD device and the HDD device, and combines the address redirection and small block write aggregation characteristics of the storage system to improve the performance of the storage system.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular relates to a data storage method and device based on address redirection. Background Art

[0002] In the data storage field, storage systems typically include hard disk drives (HDDs) and solid-state drives (SSDs). SSDs offer faster read and write speeds, but are more expensive and have smaller capacities. HDDs offer larger capacities and lower costs, but slower read and write speeds. Therefore, a combination of these two options is often employed to leverage the high speed of SSDs and the large capacity of HDDs, creating a cost-effective storage system.

[0003] In existing technologies, a layer of cache is typically built on top of an HDD using an SSD, and multiple cache strategies are provided to leverage the high-speed characteristics of the SSD to improve the HDD's IO speed. Most cache solutions (e.g., Bache and Ceph) support multiple cache strategies between SSDs and HDDs. Commonly used strategies include: writeback, which writes all data to the SSD first and then waits for the system to write the data back to the HDD; writethrough, which writes data to both the SSD and HDD simultaneously; and writearound, which writes data directly to the HDD. In other cache solutions, algorithms are used to detect access hotspots based on parameters such as data access frequency and data access time. Data with a hotspot value exceeding a threshold is stored in the SSD, while data with a hotspot value below a threshold is stored in the HDD.

[0004] However, the above scheme does not fully utilize the advantages of both disks and avoid their disadvantages: First, the SSD lifespan is limited by the number of erase cycles, while in traditional caching schemes, all data must first pass through the SSD and then be flushed to the HDD; Second, there is asymmetric performance between large and small blocks, with large block operations performing much better than small block operations. In traditional caching schemes, random small block writes remain random small block writes whether entering the SSD or going from the SSD to the HDD, which seriously affects performance; Third, traditional caching schemes do not fully consider the current system load when performing I / O scheduling. Summary of the Invention

[0005] One of the objectives of the present invention is to provide a data storage method based on address redirection, which utilizes the characteristics of the disk and the current load conditions to perform IO scheduling between SSD devices and HDD devices, and combines the address redirection and small block write aggregation characteristics of the storage system to improve the performance of the storage system.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A data storage method based on address redirection is used to store data in a storage object, wherein the storage object is a combination of an HDD device and an SSD device. The data storage method based on address redirection includes:

[0008] Step 1: Receive a write request, extract the data block identifier, data offset, and data length in the write request, and determine whether this is a large block write or a small block write based on the data length. Query the metadata database based on the data block identifier and data offset to determine whether this is a new write operation or an overwrite write operation.

[0009] Step 2: Obtain the bandwidth, IOPS, and latency of the HDD device within the current sliding time window to determine the load condition of the HDD device, where the load condition of the HDD device includes light load, medium load, and heavy load;

[0010] Step 3: Write data based on the load of the HDD device, including:

[0011] If the HDD device is lightly loaded, the data specified by the write request is written to the HDD device;

[0012] If the HDD device is moderately loaded, then when this is a large block write, the data specified by the write request is written to the HDD device; when this is a small block write, the data specified by the write request is written to the SSD device;

[0013] If the HDD device is heavily loaded, the data specified by the write request is written to the SSD device;

[0014] When data is written to an HDD device, if this is a new write operation, the data is directly written to the HDD device and data distribution information is generated. If this is an overwrite write operation, the data is written to the HDD device based on address redirection and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, the space is directly released; if it is located in the SSD device, it is marked as invalid.

[0015] When data is written to an SSD device, if this is a new write operation, the pre-allocated free blocks are selected for appending and data distribution information is generated. If this is an overwrite write operation, the free blocks allocated based on address redirection are written and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, it is marked as old data, and if it is located in the SSD device, it is marked as invalid.

[0016] Step 4: Persist the data distribution information and address space usage information in the metadata database to complete data storage.

[0017] Several optional methods are also provided below, but they are not intended to be additional limitations on the above-mentioned overall solution. They are merely further supplements or optimizations. Under the premise that there are no technical or logical contradictions, each optional method can be combined separately for the above-mentioned overall solution, or multiple optional methods can be combined.

[0018] Preferably, determining the load condition of the HDD device includes:

[0019] The three situations where bandwidth is greater than or equal to the first set threshold, IOPS is greater than or equal to the second set threshold, and latency is greater than or equal to the third set threshold are all at the first level;

[0020] The three situations where bandwidth is greater than or equal to the fourth set threshold and less than the first set threshold, IOPS is greater than or equal to the fifth set threshold and less than the second set threshold, and latency is greater than or equal to the sixth set threshold and less than the third set threshold are all rated as second level.

[0021] The three situations where bandwidth is less than the fourth set threshold, IOPS is less than the fifth set threshold, and latency is less than the sixth set threshold are all at the third level;

[0022] If the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 1, the load condition of the HDD device is heavily loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 2, the load condition of the HDD device is moderately loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 3, the load condition of the HDD device is lightly loaded.

[0023] Preferably, the data storage method based on address redirection further includes: flushing the data in the SSD device to the HDD device according to the capacity of the SSD device, specifically including the following steps:

[0024] Step a: obtaining the capacity usage of the SSD device, and determining, based on the capacity usage, whether the capacity usage of the SSD device is slightly used, partially used, or mostly used;

[0025] Step b: Flash data based on the capacity usage of the SSD device, including:

[0026] If the SSD device's capacity is only partially used, data will not be flushed;

[0027] If the capacity of the SSD device is partially used, data is flushed based on the load of the HDD device and data distribution information is generated;

[0028] If the capacity of the SSD device is mostly used, the data in the free blocks of the SSD device is sorted and flushed to the HDD device as a whole, and data distribution information is generated;

[0029] Step c: persist the data distribution information and address space usage information to the metadata database to complete the data refresh.

[0030] Preferably, the step of refreshing data according to the load of the HDD device includes:

[0031] If the HDD device is lightly loaded, the data written by the overwrite operation in the free blocks of the SDD device is directly flushed to the original location of the data in the HDD device, and the data written by the new write operation in the free blocks of the SDD device is sorted and flushed to the HDD device as a whole block;

[0032] If the HDD device is moderately loaded or heavily loaded, data in the free blocks of the SSD device is cleaned up and flushed to the HDD device in its entirety.

[0033] Preferably, the data arrangement of the free blocks in the SSD device includes:

[0034] 1) discard the data fragments marked as invalid in the free block;

[0035] 2) For the data fragments in the free blocks, they are merged according to the data identifiers, and the data fragments with the same data identifiers are sorted and spliced according to the logical addresses.

[0036] The present invention provides a data storage method based on address redirection, which fully utilizes the characteristics of the disk and can perform IO scheduling between SSDs and HDDs according to the current load conditions. It also combines the address redirection and small block write aggregation characteristics of the storage system to improve the performance of the storage system while ensuring the atomicity of write transactions.

[0037] The second purpose of the present invention is to provide a data storage device based on address redirection, which utilizes the characteristics of the disk and the current load conditions to perform IO scheduling between SSD devices and HDD devices, and combines the address redirection and small block write aggregation characteristics of the storage system to improve the performance of the storage system.

[0038] To achieve the above object, the technical solution adopted by the present invention is:

[0039] A data storage device based on address redirection is used to store data in a storage object, wherein the storage object is a combination of an HDD device and an SSD device, and the data storage device based on address redirection includes:

[0040] A first processing module is configured to receive a write request, extract a data block identifier, a data offset, and a data length from the write request, determine whether this is a large block write or a small block write based on the data length, and query a metadata database based on the data block identifier and the data offset to determine whether this is a new write operation or an overwrite write operation;

[0041] The second processing module is configured to obtain the bandwidth, IOPS, and latency of the HDD device within the current sliding time window and determine the load condition of the HDD device, where the load condition of the HDD device includes light load, medium load, and heavy load;

[0042] The third processing module is used to write data according to the load of the HDD device and perform the following operations:

[0043] If the HDD device is lightly loaded, the data specified by the write request is written to the HDD device;

[0044] If the HDD device is moderately loaded, then when this is a large block write, the data specified by the write request is written to the HDD device; when this is a small block write, the data specified by the write request is written to the SSD device;

[0045] If the HDD device is heavily loaded, the data specified by the write request is written to the SSD device;

[0046] When data is written to an HDD device, if this is a new write operation, the data is directly written to the HDD device and data distribution information is generated. If this is an overwrite write operation, the data is written to the HDD device based on address redirection and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, the space is directly released; if it is located in the SSD device, it is marked as invalid.

[0047] When data is written to an SSD device, if this is a new write operation, the pre-allocated free blocks are selected for appending and data distribution information is generated. If this is an overwrite write operation, the free blocks allocated based on address redirection are written and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, it is marked as old data, and if it is located in the SSD device, it is marked as invalid.

[0048] The fourth processing module is used to persist the data distribution information and address space usage information to the metadata database to complete data storage.

[0049] Preferably, the determining of the load condition of the HDD device comprises performing the following operations:

[0050] The three situations where bandwidth is greater than or equal to the first set threshold, IOPS is greater than or equal to the second set threshold, and latency is greater than or equal to the third set threshold are all at the first level;

[0051] The three situations where bandwidth is greater than or equal to the fourth set threshold and less than the first set threshold, IOPS is greater than or equal to the fifth set threshold and less than the second set threshold, and latency is greater than or equal to the sixth set threshold and less than the third set threshold are all rated as second level.

[0052] The three situations where bandwidth is less than the fourth set threshold, IOPS is less than the fifth set threshold, and latency is less than the sixth set threshold are all at the third level;

[0053] If the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 1, the load condition of the HDD device is heavily loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 2, the load condition of the HDD device is moderately loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 3, the load condition of the HDD device is lightly loaded.

[0054] Preferably, the data storage device based on address redirection further includes a fifth processing module, configured to flush the data in the SSD device to the HDD device according to the capacity of the SSD device, and perform the following operations:

[0055] Step a: obtaining the capacity usage of the SSD device, and determining, based on the capacity usage, whether the capacity usage of the SSD device is slightly used, partially used, or mostly used;

[0056] Step b: Flash data based on the capacity usage of the SSD device, including:

[0057] If the SSD device's capacity is only partially used, data will not be flushed;

[0058] If the capacity of the SSD device is partially used, data is flushed based on the load of the HDD device and data distribution information is generated;

[0059] If the capacity of the SSD device is mostly used, the data in the free blocks of the SSD device is sorted and flushed to the HDD device as a whole, and data distribution information is generated;

[0060] Step c: persist the data distribution information and address space usage information to the metadata database to complete the data refresh.

[0061] Preferably, the data is flushed according to the load of the HDD device, and the following operations are performed:

[0062] If the HDD device is lightly loaded, the data written by the overwrite operation in the free blocks of the SDD device is directly flushed to the original location of the data in the HDD device, and the data written by the new write operation in the free blocks of the SDD device is sorted and flushed to the HDD device as a whole block;

[0063] If the HDD device is moderately loaded or heavily loaded, data in the free blocks of the SSD device is cleaned up and flushed to the HDD device in its entirety.

[0064] Preferably, the data arrangement for the free blocks in the SSD device is performed by performing the following operations:

[0065] 1) discard the data fragments marked as invalid in the free block;

[0066] 2) For the data fragments in the free blocks, they are merged according to the data identifiers, and the data fragments with the same data identifiers are sorted and spliced according to the logical addresses. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Flowchart of the data storage method based on address redirection of the present invention;

[0068] Figure 2 This is a flow chart of data flushing in the data storage method based on address redirection of the present invention;

[0069] Figure 3 This is a schematic diagram of free block data arrangement in the present invention;

[0070] Figure 4 A schematic diagram of executing data writing in a data storage device based on address redirection according to the present invention;

[0071] Figure 5 A schematic diagram of executing data flushing of a data storage device based on address redirection according to the present invention;

[0072] Figure 6 This is a data distribution diagram provided by the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0075] Current cache systems do not fully consider the advantages and disadvantages of SSDs (SDDs) and HDDs (HDDs) as well as the current system load. Typically, all data passes through the SSD first and then is flushed to the HDD, which is detrimental to the limited lifespan of SSDs. Furthermore, random small-block writes are not aggregated. Random small-block writes remain random small-block writes whether entering the SSD or going from the SSD to the HDD, severely impacting performance.

[0076] In view of the above, this embodiment provides a data storage method based on address redirection, which fully utilizes the characteristics of the disk and can perform IO scheduling between SSDs and HDDs according to the current load conditions. It also combines the address redirection and small block write aggregation characteristics of the storage system to improve the performance of the storage system while ensuring the atomicity of write transactions.

[0077] like Figure 1 As shown, the data storage method based on address redirection of this embodiment includes:

[0078] Step 1: Receive a write request, extract the data block identifier, data offset, and data length in the write request, and determine whether this is a large block write or a small block write based on the data length. Query the metadata database based on the data block identifier and data offset to determine whether this is a new write operation or an overwrite write operation.

[0079] After receiving the user's data write request, the relevant information in the write request is parsed and obtained, namely the data block identifier, data offset and data length.

[0080] When determining whether this is a large block write or a small block write, the judgment is made based on the data length. If the data length is greater than the data amount threshold, it is determined that this is a large block write; otherwise, it is determined that this is a small block write.

[0081] When determining whether this is a new write operation or an overwrite write operation, the metadata database is queried according to the data block identifier to obtain metadata corresponding to the data block identifier, and the metadata is parsed to confirm whether this is a new write operation or an overwrite write operation.

[0082] Step 2: Obtain the bandwidth, IOPS, and latency of the HDD device within the current sliding time window to determine the load condition of the HDD device. The load condition of the HDD device includes light load, medium load, and heavy load.

[0083] This embodiment comprehensively considers the HDD device's bandwidth, IOPS, and latency to determine the HDD device load from multiple perspectives, avoiding the inaccurate results of a single factor. This embodiment uses the highest load level reflected by bandwidth, IOPS, and latency as the final result. The specific judgment logic is as follows.

[0084] The three situations of bandwidth being greater than or equal to the first set threshold, IOPS being greater than or equal to the second set threshold, and latency being greater than or equal to the third set threshold are all considered the first level.

[0085] The three situations where bandwidth is greater than or equal to the fourth set threshold and less than the first set threshold, IOPS is greater than or equal to the fifth set threshold and less than the second set threshold, and latency is greater than or equal to the sixth set threshold and less than the third set threshold are all second level.

[0086] The three situations where the bandwidth is less than the fourth set threshold, the IOPS is less than the fifth set threshold, and the latency is less than the sixth set threshold are all considered the third level.

[0087] If the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 1, the load condition of the HDD device is heavily loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 2, the load condition of the HDD device is moderately loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 3, the load condition of the HDD device is lightly loaded.

[0088] This embodiment first defines three levels, each of which indicates the satisfaction of bandwidth, IOPS, and latency. Then, based on the actual bandwidth, IOPS, and latency of the HDD device, the highest level met is used as the basis for determining the load condition of the HDD device. This not only integrates the three factors, but also simply and clearly determines the load condition of the HDD device.

[0089] This embodiment controls the data writing object according to the load condition of the HDD device, fully utilizes the characteristics of the disk and the current load condition, and performs IO scheduling between the SSD and the HDD.

[0090] Step 3: Write data based on the load of the HDD device, including:

[0091] If the HDD device is lightly loaded, the data specified by the write request is written to the HDD device.

[0092] If the HDD device is moderately loaded, when this is a large block write, the data specified by the write request is written to the HDD device; when this is a small block write, the data specified by the write request is written to the SSD device.

[0093] If the HDD device is heavily loaded, the data specified by the write request is written to the SSD device.

[0094] Among them, when data is written to the HDD device, if this is a new write operation, the data is directly written to the HDD device and data distribution information (including data storage address, data block and sector, etc.) is generated. If this is an overwrite write operation, the data is written to the HDD device based on address redirection and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, the space is directly released. If it is located in the SSD device, it is marked as invalid. When data is written to the SSD device, if this is a new write operation, the pre-allocated free block (Appending block) is selected for appending and data distribution information is generated. If this is an overwrite write operation, the free block allocated based on address redirection is written and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, it is marked as old data. If it is located in the SSD device, it is marked as invalid.

[0095] In this embodiment, the overwritten address control refers to the address space initially pointed to. For example, for data block A, the contents of (0-512) are first written and stored in the 1024th sector of the HDD. Then, the contents of (0-512) are modified and written to the 1096th sector of the HDD. The overwritten address space here refers to the 1024th sector.

[0096] Step 4: Persist the data distribution information and address space usage information (mainly the address space usage status) in the metadata database to complete the data storage. A data write success message is returned to the user.

[0097] Since the capacity of the SDD is relatively small, the data storage method based on address redirection in this embodiment also needs to regularly flush the data in the SDD to the HDD device. Figure 2 As shown, this embodiment flushes data from the SSD device to the HDD device according to the capacity of the SSD device, specifically including the following steps:

[0098] Step a: obtaining the capacity usage of the SSD device, and determining, based on the capacity usage, whether the capacity usage of the SSD device is slightly used, partially used, or mostly used.

[0099] When judging the capacity usage of the SDD, if the capacity usage of the SSD device exceeds the eighth set threshold, it is determined that the capacity usage of the SSD device is mostly used; if the capacity usage of the SSD device exceeds the ninth set threshold but does not exceed the eighth set threshold (the eighth set threshold is greater than the ninth set threshold), it is determined that the capacity usage of the SSD device is partially used; if the capacity usage of the SSD device does not exceed the ninth set threshold, it is determined that the capacity usage of the SSD device is slightly used.

[0100] Step b: Flash data based on the capacity usage of the SSD device, including:

[0101] 1) If the capacity of the SSD device is only partially used, data is not flushed and the SSD space is reclaimed.

[0102] 2) If the capacity of the SSD device is partially used, the storage device will flush the data in the SSD to the HDD at a slightly slower speed, flush the data according to the load of the HDD device, and generate data distribution information.

[0103] In this case, when data is flushed, if the HDD device is lightly loaded, the data written by the overwrite operation in the free block of the SDD device is directly flushed to the original location of the data in the HDD device (no space allocation and address redirection are required. For example, for data block A, part of the content (0-512) is stored in the 1024th sector of the HDD. When the other part of the content of data block A written by the overwrite operation needs to be flushed to the HDD, it is directly flushed to the 1024th sector of the HDD). The data written by the new write operation in the free block of the SDD device is sorted and flushed to the HDD device as a whole. If the HDD device is moderately loaded or heavily loaded, the data in the free block of the SSD device is sorted and flushed to the HDD device as a whole.

[0104] 3) If the SSD capacity is mostly used, the storage device will flush the data from the SSD to the HDD at a faster speed. The free blocks in the SSD will be sorted and flushed to the HDD in their entirety, generating data distribution information.

[0105] In this case, when the entire block of data is flushed to the HDD device, for the data written by the overwrite write operation, address redirection is performed after the flush write is completed, that is, address redirection settings are performed in the address space usage information of step c.

[0106] Step c: persist the data distribution information and address space usage information to the metadata database to complete the data flush. In this embodiment, the data sorting process for the free blocks in the SSD device is as follows: Figure 3As shown, it includes: 1) discarding data fragments marked as invalid in the free blocks; 2) merging the data fragments in the free blocks according to data identifiers, and sorting and splicing the data fragments with the same data identifier according to the logical addresses.

[0107] This embodiment persists the object distribution and disk usage to the metadata database, and the completion of metadata persistence marks the end of the write request. Figure 4 As shown, the same data segment may have two address pointers in the metadata database, one pointing to the SSD and the other pointing to the HDD, where the SSD contains the newer data segment.

[0108] The data storage method provided in this embodiment performs I / O scheduling between the HDD and SSD based on the storage device load. When the system load is low, write requests are routed to the HDD. Under moderate load, large-block writes are routed to the SSD, while small-block writes are routed to the HDD. Under heavy load, write requests are routed to the SSD. This maximizes system performance while extending the life of the SSD. Furthermore, write requests are aggregated into sequential writes in the SSD and written to the HDD in large blocks during flushing, leveraging the fact that the sequential write performance of the SSD and the large-block write performance of the HDD are far superior to random writes and small-block writes.

[0109] In another embodiment, a data storage device based on address redirection is provided for storing data in a storage object, wherein the storage object is a combination of an HDD device and an SSD device. The data storage device based on address redirection includes:

[0110] A first processing module is configured to receive a write request, extract a data block identifier, a data offset, and a data length from the write request, determine whether this is a large block write or a small block write based on the data length, and query a metadata database based on the data block identifier and the data offset to determine whether this is a new write operation or an overwrite write operation;

[0111] The second processing module is configured to obtain the bandwidth, IOPS, and latency of the HDD device within the current sliding time window and determine the load condition of the HDD device, where the load condition of the HDD device includes light load, medium load, and heavy load;

[0112] The third processing module is used to write data according to the load of the HDD device and perform the following operations:

[0113] If the HDD device is lightly loaded, the data specified by the write request is written to the HDD device;

[0114] If the HDD device is moderately loaded, then when this is a large block write, the data specified by the write request is written to the HDD device; when this is a small block write, the data specified by the write request is written to the SSD device;

[0115] If the HDD device is heavily loaded, the data specified by the write request is written to the SSD device;

[0116] When data is written to an HDD device, if this is a new write operation, the data is directly written to the HDD device and data distribution information is generated. If this is an overwrite write operation, the data is written to the HDD device based on address redirection and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, the space is directly released; if it is located in the SSD device, it is marked as invalid.

[0117] When data is written to an SSD device, if this is a new write operation, the pre-allocated free blocks are selected for appending and data distribution information is generated. If this is an overwrite write operation, the free blocks allocated based on address redirection are written and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, it is marked as old data, and if it is located in the SSD device, it is marked as invalid.

[0118] The fourth processing module is used to persist the data distribution information and address space usage information to the metadata database to complete data storage.

[0119] In a specific embodiment, the determining of the load condition of the HDD device includes performing the following operations:

[0120] The three situations where bandwidth is greater than or equal to the first set threshold, IOPS is greater than or equal to the second set threshold, and latency is greater than or equal to the third set threshold are all at the first level;

[0121] The three situations where bandwidth is greater than or equal to the fourth set threshold and less than the first set threshold, IOPS is greater than or equal to the fifth set threshold and less than the second set threshold, and latency is greater than or equal to the sixth set threshold and less than the third set threshold are all rated as second level.

[0122] The three situations where bandwidth is less than the fourth set threshold, IOPS is less than the fifth set threshold, and latency is less than the sixth set threshold are all at the third level;

[0123] If the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 1, the load condition of the HDD device is heavily loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 2, the load condition of the HDD device is moderately loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 3, the load condition of the HDD device is lightly loaded.

[0124] In a specific embodiment, the data storage device based on address redirection further includes a fifth processing module configured to flush the data in the SSD to the HDD device according to the capacity of the SSD device, and perform the following operations:

[0125] Step a: obtaining the capacity usage of the SSD device, and determining, based on the capacity usage, whether the capacity usage of the SSD device is slightly used, partially used, or mostly used;

[0126] Step b: Flash data based on the capacity usage of the SSD device, including:

[0127] If the SSD device's capacity is only partially used, data will not be flushed;

[0128] If the capacity of the SSD device is partially used, data is flushed based on the load of the HDD device and data distribution information is generated;

[0129] If the capacity of the SSD device is mostly used, the data in the free blocks of the SSD device is sorted and flushed to the HDD device as a whole, and data distribution information is generated;

[0130] Step c: persist the data distribution information and address space usage information to the metadata database to complete the data refresh.

[0131] In a specific embodiment, the data is flushed according to the load of the HDD device, and the following operations are performed:

[0132] If the HDD device is lightly loaded, the data written by the overwrite operation in the free blocks of the SDD device is directly flushed to the original location of the data in the HDD device, and the data written by the new write operation in the free blocks of the SDD device is written as a whole;

[0133] If the HDD device is moderately loaded or heavily loaded, data in the free blocks of the SSD device is cleaned up and flushed to the HDD device in its entirety.

[0134] In a specific embodiment, the data arrangement of the free blocks in the SSD device is performed by performing the following operations:

[0135] 1) discard the data fragments marked as invalid in the free block;

[0136] 2) For the data fragments in the free blocks, they are merged according to the data identifiers, and the data fragments with the same data identifiers are sorted and spliced according to the logical addresses.

[0137] It should be noted that the specific limitations of the data storage device based on address redirection can be found in the aforementioned limitations of the data storage method based on address redirection, and will not be further elaborated here. Furthermore, the first to fifth processing modules described above are merely module distribution methods in one embodiment. In other embodiments, the modules may be further divided based on their functions.

[0138] For example, Figures 5-6 As shown, in another embodiment, a data storage device based on address redirection is provided, comprising:

[0139] The performance monitoring module collects statistics and monitors HDD request information. It uses a sliding time window to collect statistics on operations within the time window. Key statistics include data read and write length, data read and write times, and request latency. Statistics slide as the time window slides.

[0140] The data write management module determines whether the HDD device is heavily, moderately, or lightly loaded based on the monitoring and statistical information from the performance monitoring module. It determines whether written data is sent to the HDD or SSD based on the HDD load and data block size. If the HDD is heavily loaded, all write requests are sent to the SSD. If the HDD is moderately loaded, large block writes are sent to the HDD, and small block writes are sent to the SSD. If the HDD is lightly loaded, all write requests are sent to the HDD.

[0141] The data reading management module is responsible for data addressing. It mainly follows up on the data identifier to query the data distribution location from the metadata database, reads it, performs some sort of organization, and then returns it.

[0142] The SSD space management module manages SSD space and persists space allocation information in the metadata database. This module pre-creates appending blocks, to which data arriving on the SSD is sequentially appended. When the size of an appending block reaches the agreed upper limit, the SSD creates a new appending block.

[0143] The SSD cache flush module is used to organize and flush the appending blocks according to the SSD space situation. The flushing strategy is selected based on the SSD capacity usage and HDD load. When most of the SSD capacity has been used, the appending blocks are flushed as a whole at a relatively fast speed; when part of the capacity has been used, the appending blocks are flushed at a slightly slower speed. If the HDD load is light, the overwrite is directly flushed to the original location. If the HDD load is high or has a certain load, the entire block is flushed. After the appending block is flushed, the SSD space management module is notified to release the corresponding space.

[0144] The HDD space management module manages HDD space and persists space allocation information in the metadata database. It allocates space in response to requests from the data write management module and the SSD cache flush module.

[0145] The metadata database stores metadata information about storage devices, primarily including data distribution and SSD and HDD space usage. When the storage device boots up, other modules load the space usage and data distribution information from the metadata database to initialize the storage device. When processing a write request, after data is flushed to the SSD or HDD according to the schedule, the relevant metadata is persisted to the metadata database, marking the end of the write process. When reading data, the data is retrieved based on its distribution in the metadata database, sorted, and returned.

[0146] The data storage device based on address redirection provided by the embodiment of the present invention performs I / O scheduling between SSDs and HDDs according to disk load and aggregates write operations through the SSD, thereby fully utilizing system performance while extending the life of the SSD as much as possible.

[0147] The present invention provides a data storage method and device based on address redirection. These methods and devices fully utilize the characteristics of disks and the current load to perform I / O scheduling between SSDs and HDDs. Furthermore, they combine the address redirection and small-block write aggregation features of the storage system to improve storage system performance. The data storage device of the present invention can function as a standalone storage device or as a storage node in a distributed storage system. This embodiment also provides a distributed data storage system comprising two or more of the aforementioned storage devices.

[0148] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0149] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A data storage method based on address redirection, for storing data in a storage object, wherein the storage object is a combination of an HDD device and an SSD device, characterized in that: The data storage method based on address redirection includes: Step 1: Receive a write request, extract the data block identifier, data offset, and data length in the write request, and determine whether this is a large block write or a small block write based on the data length. Query the metadata database based on the data block identifier and data offset to determine whether this is a new write operation or an overwrite write operation. Step 2: Obtain the bandwidth, IOPS, and latency of the HDD device within the current sliding time window to determine the load condition of the HDD device, where the load condition of the HDD device includes light load, medium load, and heavy load; Step 3: Write data based on the load of the HDD device, including: If the HDD device is lightly loaded, the data specified by the write request is written to the HDD device; If the HDD device is moderately loaded, then when this is a large block write, the data specified by the write request is written to the HDD device; when this is a small block write, the data specified by the write request is written to the SSD device; If the HDD device is heavily loaded, the data specified by the write request is written to the SSD device; When data is written to an HDD device, if this is a new write operation, the data is directly written to the HDD device and data distribution information is generated. If this is an overwrite write operation, the data is written to the HDD device based on address redirection and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, the space is directly released; if it is located in the SSD device, it is marked as invalid. When data is written to an SSD device, if this is a new write operation, the pre-allocated free blocks are selected for appending and data distribution information is generated. If this is an overwrite write operation, the free blocks allocated based on address redirection are written and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, it is marked as old data, and if it is located in the SSD device, it is marked as invalid. Step 4: Persist the data distribution information and address space usage information in the metadata database to complete data storage.

2. The data storage method based on address redirection according to claim 1, characterized in that: Determining the load condition of the HDD device includes: The three situations where bandwidth is greater than or equal to the first set threshold, IOPS is greater than or equal to the second set threshold, and latency is greater than or equal to the third set threshold are all at the first level; The three situations where bandwidth is greater than or equal to the fourth set threshold and less than the first set threshold, IOPS is greater than or equal to the fifth set threshold and less than the second set threshold, and latency is greater than or equal to the sixth set threshold and less than the third set threshold are all rated as second level. The three situations where bandwidth is less than the fourth set threshold, IOPS is less than the fifth set threshold, and latency is less than the sixth set threshold are all at the third level; If the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 1, the load condition of the HDD device is heavily loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 2, the load condition of the HDD device is moderately loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 3, the load condition of the HDD device is lightly loaded.

3. The data storage method based on address redirection according to claim 1, characterized in that: The data storage method based on address redirection further includes: flushing the data in the SSD device to the HDD device according to the capacity of the SSD device, specifically including the following steps: Step a: obtaining the capacity usage of the SSD device, and determining, based on the capacity usage, whether the capacity usage of the SSD device is slightly used, partially used, or mostly used; Step b: Flash data based on the capacity usage of the SSD device, including: If the SSD device's capacity is only partially used, data will not be flushed; If the capacity of the SSD device is partially used, data is flushed based on the load of the HDD device and data distribution information is generated; If the capacity of the SSD device is mostly used, the data in the free blocks of the SSD device is sorted and flushed to the HDD device as a whole, and data distribution information is generated; Step c: persist the data distribution information and address space usage information to the metadata database to complete the data refresh.

4. The data storage method based on address redirection according to claim 3, characterized in that: The data flushing according to the load of the HDD device includes: If the HDD device is lightly loaded, the data written by the overwrite operation in the free blocks of the SDD device is directly flushed to the original location of the data in the HDD device, and the data written by the new write operation in the free blocks of the SDD device is sorted and flushed to the HDD device as a whole block; If the HDD device is moderately loaded or heavily loaded, data in the free blocks of the SSD device is cleaned up and flushed to the HDD device in its entirety.

5. The data storage method based on address redirection according to claim 3 or 4, characterized in that: The data arrangement of the free blocks in the SSD device includes: 1) discard the data fragments marked as invalid in the free block; 2) For the data fragments in the free blocks, they are merged according to the data identifiers, and the data fragments with the same data identifiers are sorted and spliced according to the logical addresses.

6. A data storage device based on address redirection, for storing data in a storage object, wherein the storage object is a combination of an HDD device and an SSD device, characterized in that: The data storage device based on address redirection includes: A first processing module is configured to receive a write request, extract a data block identifier, a data offset, and a data length from the write request, determine whether this is a large block write or a small block write based on the data length, and query a metadata database based on the data block identifier and the data offset to determine whether this is a new write operation or an overwrite write operation; The second processing module is configured to obtain the bandwidth, IOPS, and latency of the HDD device within the current sliding time window and determine the load condition of the HDD device, where the load condition of the HDD device includes light load, medium load, and heavy load; The third processing module is used to write data according to the load of the HDD device and perform the following operations: If the HDD device is lightly loaded, the data specified by the write request is written to the HDD device; If the HDD device is moderately loaded, then when this is a large block write, the data specified by the write request is written to the HDD device; when this is a small block write, the data specified by the write request is written to the SSD device; If the HDD device is heavily loaded, the data specified by the write request is written to the SSD device; When data is written to an HDD device, if this is a new write operation, the data is directly written to the HDD device and data distribution information is generated. If this is an overwrite write operation, the data is written to the HDD device based on address redirection and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, the space is directly released; if it is located in the SSD device, it is marked as invalid. When data is written to an SSD device, if this is a new write operation, the pre-allocated free blocks are selected for appending and data distribution information is generated. If this is an overwrite write operation, the free blocks allocated based on address redirection are written and data distribution information is generated. For the overwritten address space, if it is located in the HDD device, it is marked as old data, and if it is located in the SSD device, it is marked as invalid. The fourth processing module is used to persist the data distribution information and address space usage information to the metadata database to complete data storage.

7. The data storage device based on address redirection according to claim 6, characterized in that: Determining the load condition of the HDD device involves performing the following operations: The three situations where bandwidth is greater than or equal to the first set threshold, IOPS is greater than or equal to the second set threshold, and latency is greater than or equal to the third set threshold are all at the first level; The three situations where bandwidth is greater than or equal to the fourth set threshold and less than the first set threshold, IOPS is greater than or equal to the fifth set threshold and less than the second set threshold, and latency is greater than or equal to the sixth set threshold and less than the third set threshold are all rated as second level. The three situations where bandwidth is less than the fourth set threshold, IOPS is less than the fifth set threshold, and latency is less than the sixth set threshold are all at the third level; If the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 1, the load condition of the HDD device is heavily loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 2, the load condition of the HDD device is moderately loaded; if the highest level of bandwidth, IOPS, and latency that the HDD device meets within the current sliding time window is level 3, the load condition of the HDD device is lightly loaded.

8. The data storage device based on address redirection according to claim 6, characterized in that: The data storage device based on address redirection further includes a fifth processing module, configured to flush the data in the SSD device to the HDD device according to the capacity of the SSD device, and perform the following operations: Step a: obtaining the capacity usage of the SSD device, and determining, based on the capacity usage, whether the capacity usage of the SSD device is slightly used, partially used, or mostly used; Step b: Flash data based on the capacity usage of the SSD device, including: If the SSD device's capacity is only partially used, data will not be flushed; If the capacity of the SSD device is partially used, data is flushed based on the load of the HDD device and data distribution information is generated; If the capacity of the SSD device is mostly used, the data in the free blocks of the SSD device is sorted and flushed to the HDD device as a whole, and data distribution information is generated; Step c: persist the data distribution information and address space usage information to the metadata database to complete the data refresh.

9. The data storage device based on address redirection according to claim 8, characterized in that: The data is flushed according to the load of the HDD device, and the following operations are performed: If the HDD device is lightly loaded, the data written by the overwrite operation in the free blocks of the SDD device is directly flushed to the original location of the data in the HDD device, and the data written by the new write operation in the free blocks of the SDD device is sorted and flushed to the HDD device as a whole block; If the HDD device is moderately loaded or heavily loaded, data in the free blocks of the SSD device is cleaned up and flushed to the HDD device in its entirety.

10. The data storage device based on address redirection according to claim 8 or 9, characterized in that: The data arrangement of the free blocks in the SSD device is performed by performing the following operations: 1) discard the data fragments marked as invalid in the free block; 2) For the data fragments in the free blocks, they are merged according to the data identifiers, and the data fragments with the same data identifiers are sorted and spliced according to the logical addresses.

Images