Metadata management method and device based on online deduplication function

Abstract

The invention relates to the technical field of computers, and provides a metadata management method and device based on an online deduplication function, and the method comprises the steps: receiving a metadata I / O request sent by a storage controller in storage equipment, in metadata, an identifier of a data block is an identifier key, a logic address block of the data block is a logic address key, and the logic address key is a logic address key; identifying physical address blocks of the data blocks corresponding to the logic address blocks as values; under the condition that the metadata I / O request is a write request, the metadata I / O request comprises an identification key and a first logic address key in metadata, determining a to-be-written key value pair based on the identification key and the first logic address key, and writing the to-be-written key value pair into a nonvolatile memory of a key value disk, and recording the keys in the to-be-written key value pair and the address index of the storage address of the to-be-written key value pair in the nonvolatile memory in the volatile memory of the key value disk. According to the invention, the complexity of the metadata management process design on the storage controller is reduced, and the consumption of resources in the storage device is reduced.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a metadata management method and device based on online deduplication function. Background Art

[0002] The Inline Deduplication function means that when the target data block is stored on the storage device, the deduplication process is run. If the storage device has stored a data block identical to the target data block, that is, the data is duplicated, the target data block will not be written to the disk, but will be deleted before it is written to the disk, thereby ensuring that only one copy of the same data is stored on the storage device to save storage space.

[0003] In order to find and determine whether the data blocks are the same data blocks, each data block will maintain a set of metadata, which includes the mapping relationship between the data block's fingerprint (i.e., the unique identifier of the data block, such as: hash value) and the physical address block (Physical Block Address, PBA) storing the data block, and the mapping relationship between the logical address block (Logical Block Address, LBA) of the data block and the physical address block. At present, the management of metadata is implemented by (multiple) controllers of the storage device, and in order to quickly access the metadata, the metadata is cached in the memory of the storage device. However, with the sharp increase in the amount of user data, part of the metadata must also be stored in a large-capacity disk. Therefore, each time the online deduplication function is executed, the metadata must be mirrored (metadata mirroring between multiple controllers), power-off protection, and cache data organization structure conversion. This series of operations makes the metadata management logic very complex and consumes more CPU and memory resources of the storage device. Summary of the invention

[0004] The present invention provides a metadata management method and device based on online deduplication function, which are used to solve the problem that the metadata management logic of the controller in the existing storage device is complex and consumes more CPU and memory resources of the storage device.

[0005] The present invention provides a metadata management method based on online deduplication function.

[0006] Receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value; In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into a non-volatile memory of a key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0007] According to a metadata management method based on online deduplication function provided by the present invention, based on the identification key and the first logical address key, the key-value pair to be written is determined, and the key-value pair to be written is written into the non-volatile memory of the key-value disk, including the following steps.

[0008] The identification key is looked up in a record in the volatile memory of the key-value disk.

[0009] When the identification key is found, the value corresponding to the identification key is determined to be the target value, the key-value pair to be written is formed with the first logical address key and the target value, and the key-value pair to be written is written into the non-volatile memory of the key-value disk.

[0010] If the identification key is not found, the physical address block of the data block written by the storage controller is obtained as the target value, a first key-value pair is formed with the identification key and the target value, a second key-value pair is formed with the first logical address key and the target value, it is determined that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and the key-value pairs to be written are written into the non-volatile memory of the key-value disk.

[0011] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method also includes the following steps.

[0012] The address index corresponding to the second logical address key is searched in the record of the volatile memory of the key-value disk.

[0013] The target storage address of the key-value pair to be read is obtained based on the address index corresponding to the second logical address key.

[0014] A physical address block corresponding to a second logical address key in a nonvolatile memory is read based on the target storage address, and the physical address block is returned to the storage controller.

[0015] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the method also includes the following steps.

[0016] The address index corresponding to the third logical address key is searched in the record of the volatile memory of the key-value disk.

[0017] The target storage address of the key-value pair to be deleted is obtained based on the address index corresponding to the third logical address key.

[0018] The key-value pair to be deleted is deleted from the non-volatile memory based on the target storage address.

[0019] The third logical address key and the corresponding address index are deleted from the record of the volatile memory of the key-value disk.

[0020] According to a metadata management method based on online deduplication function provided by the present invention, the non-volatile memory includes: a non-volatile memory and a non-volatile flash memory; writing the key-value pairs to be written into the non-volatile memory of the key-value disk includes: writing the key-value pairs to be written into the non-volatile memory, and when the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0021] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method also includes the following steps.

[0022] The address index corresponding to the second logical address key is searched in the record of the volatile memory of the key-value disk.

[0023] The target storage address of the key-value pair to be read is obtained based on the address index corresponding to the second logical address key.

[0024] In the case where the target storage address is the address of the non-volatile memory, a physical address block corresponding to the second logical address key in the non-volatile memory is read based on the target storage address, and the physical address block is returned to the storage controller.

[0025] When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory, and then the physical address block corresponding to the second logical address key is read from the target storage block, and the physical address block is returned to the storage controller.

[0026] According to a metadata management method based on online deduplication function provided by the present invention, when the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the method also includes the following steps.

[0027] The address index corresponding to the third logical address key is searched in the record of the volatile memory of the key-value disk.

[0028] The target storage address of the key-value pair to be deleted is obtained based on the address index corresponding to the third logical address key.

[0029] In the case where the target storage address is the address of the non-volatile memory, the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk.

[0030] In the case where the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory to the non-volatile memory, the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk; if the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0031] According to a metadata management method based on online deduplication function provided by the present invention, it also includes: when the key-value disk is started after power failure, traversing all key-value pairs in the non-volatile memory, rebuilding the address indexes of all key-value pairs, and storing the reconstructed address indexes and corresponding keys in the volatile memory.

[0032] The present invention also provides a metadata management device based on online deduplication function, comprising the following modules.

[0033] An I / O request receiving module is used to receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0034] A metadata writing module is used to, when the metadata I / O request is a write request, determine a key-value pair to be written based on the identification key and the first logical address key, write the key-value pair to be written into the non-volatile memory of the key-value disk, and record the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0035] The present invention also provides a key-value disk, comprising: an in-disk processor, a volatile memory and a non-volatile memory, wherein the in-disk processor is connected to the volatile memory and the non-volatile memory.

[0036] The non-volatile memory is used to store key-value pairs related to metadata, and the key-value pairs include: a key-value pair with an identifier of a data block as an identifier key and a physical address block of the data block as a value, and a key-value pair with a logical address block of the data block as a logical address key and a physical address block of the data block as a value.

[0037] The volatile memory is used to record the key of each key-value pair and the address index of the storage address of each key-value pair in the non-volatile memory.

[0038] The processor in the disk stores a computer program, which is used to implement the metadata management method based on the online deduplication function as described in any one of the above when executing the computer program.

[0039] The present invention also provides a storage device, comprising: a storage controller and the above-mentioned key-value disk, wherein the storage controller is connected to the on-disk processor and is used to send a metadata I / O request to the on-disk processor.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the metadata management method based on the online deduplication function as described above is implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the metadata management method based on the online deduplication function as described in any one of the above is implemented.

[0042] The metadata management method and device based on online deduplication function provided by the present invention receive a metadata I / O request sent by a storage controller in a storage device, wherein in the metadata, the identifier of a data block is an identifier key, the logical address block of the data block is a logical address key, and the physical address block of the data block corresponding to each of the identifier and the logical address block is a value; when the metadata I / O request is a write request, the metadata I / O request includes the identifier key and the first logical address key in the metadata, and based on the identifier key and the first logical address key, a key-value pair to be written is determined, the key-value pair to be written is written into a non-volatile memory of a key-value disk, and the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory are recorded in the volatile memory of the key-value disk. During the metadata write operation, the metadata key-value pairs and the corresponding address indexes are stored in the key-value disk. When the storage controller performs the online deduplication function, the metadata I / O request is sent to the key-value disk, and the key-value disk maintains and manages the metadata in the form of key-value pairs. The storage controller of the storage device no longer needs to implement processes such as metadata mirroring, power-off protection, and cache data organization structure conversion, thereby reducing the complexity of the metadata management process design on the storage controller of the storage device and reducing the consumption of CPU and memory resources in the storage device. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 This is one of the flow charts of the metadata management method based on the online deduplication function provided by the present invention.

[0045] Figure 2 It is a schematic diagram of a hardware structure for implementing the metadata management method based on the online deduplication function provided by the present invention.

[0046] Figure 3 This is the second flow chart of the metadata management method based on the online deduplication function provided by the present invention.

[0047] Figure 4 This is the third flow chart of the metadata management method based on the online deduplication function provided by the present invention.

[0048] Figure 5 This is the fourth flow chart of the metadata management method based on the online deduplication function provided by the present invention.

[0049] Figure 6 It is a structural schematic diagram of a metadata management device based on online deduplication function provided by the present invention.

[0050] Figure 7 It is a flow chart of the storage device provided by the present invention realizing the online deduplication function when writing data.

[0051] Figure 8 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] The metadata management method based on the online deduplication function of the embodiment of the present invention is as follows: Figure 1 As shown, the following steps are included.

[0054] Step S110: receiving a metadata I / O request sent by a storage controller in a storage device, wherein in the metadata, an identifier of a data block is an identifier key, a logical address block of a data block is a logical address key, and a physical address block of a data block corresponding to each of the identifier and the logical address block is a value, that is, according to a mapping between an identifier H and a physical address block P in the metadata, a key-value pair HP (key-value) from the identifier H to the physical address block P is formed, and according to a mapping between a logical address block L and a physical address block P in the metadata, a key-value pair LP from the logical address block L to the physical address block P is formed.

[0055] It should be noted that the execution subject of the metadata management method based on the online deduplication function of this embodiment is no longer the storage controller in the storage device, but a key-value disk (such as a key-value solid-state drive KV SSD), specifically, the disk processor in the key-value disk. Specifically, when the host performs an I / O operation (read, write, and delete) on a data block on the storage device, the I / O request for the data block will be sent to the storage device. When the storage device operates on the data block, it needs to perform corresponding operations on the metadata. Therefore, the storage controller will initiate a metadata I / O request to the disk processor. For example, the host needs to write a data block, and the I / O write request for the data block sent by the host includes a logical address block L and a data block D to be written. Due to the online deduplication function, after receiving the I / O write request for the data block, the storage controller calculates the identifier H (i.e., the hash value) of the data block D, and sends the metadata I / O write request including the identifier H and the logical address block L to the disk processor in the key-value disk. Specifically, the Store operation interface of the key-value disk can be called to send a metadata I / O write request. The interface details are required to be compatible with the "NVM-Express-Key-Value-Command-Set-Specification" in the NVMe2.0 standard, which defines the Store, Exist and Delete operation interfaces. In this embodiment, the Store operation interface can be implemented according to the following steps S121 to S122.

[0056] Step S120: when the metadata I / O request is a write request, perform a write operation on the metadata, the metadata I / O request includes an identification key and a first logical address key in the metadata, and specifically, the metadata write operation includes the following steps.

[0057] Step S121: Based on the identification key and the first logical address key, determine the key-value pair to be written, and write the key-value pair to be written into the non-volatile memory of the key-value disk. The key-value pair to be written is LP, or LP and HP. It is necessary to determine whether it is LP, or LP and HP based on the query result of the online deduplication function. The non-volatile memory capacity of the key-value disk is large, and it can store metadata corresponding to massive data blocks. The key-value disk is preferably a KV SSD, which has a high read and write speed, ensuring efficient operation of metadata. Figure 2 As shown, a KV SSD is set in the storage device to save and manage metadata, and the remaining SSD disks are used to store data blocks.

[0058] Step S122: Record the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk. Specifically, the key and the address index of the corresponding key-value pair are recorded as a record in the hash list for quick query. The volatile memory has a higher read and write speed. The relationship between the key and the address index of the corresponding key-value pair is recorded in the volatile memory of the key-value disk, and the address of the corresponding key-value pair can be quickly found according to the key, thereby improving the query efficiency of metadata and the efficiency of subsequent read and delete operations.

[0059] In the metadata management method based on the online deduplication function of this embodiment, since the key-value pairs of the metadata and the corresponding address indexes are stored in the key-value disk during the metadata write operation, when the storage controller executes the online deduplication function, the metadata I / O request is sent to the key-value disk, and the key-value disk maintains and manages the metadata in the form of key-value pairs. The storage controller of the storage device no longer needs to implement processes such as metadata mirroring, power-off protection, and cache data organization structure conversion, thereby reducing the complexity of the metadata management process design on the storage controller of the storage device and reducing the consumption of CPU and memory resources in the storage device.

[0060] In some embodiments, Figure 3 As shown, step S121 specifically includes the following steps.

[0061] Step S310: searching for the identification key in the record of the volatile memory of the key-value disk, that is, searching for the identification key H, to confirm whether the data block identified as H to be written has been stored in the SSD disk of the storage device.

[0062] Step S320: When the identification key is found, determine that the value corresponding to the identification key is the target value, form the key-value pair to be written with the first logical address key and the target value, and write the key-value pair to be written into the non-volatile memory of the key-value disk. Specifically, the identification key is found in the record of the volatile memory, indicating that the data block identified as H has been stored in the SSD disk of the storage device, and the HP key-value pair has been stored in the non-volatile memory of the key-value disk, and the P in the HP key-value pair is the physical address block of the data block identified as H in the SSD disk of the storage device. Obtain the address index of the key-value pair corresponding to the identification key in the record of the volatile memory, find the location of the key-value pair stored in the non-volatile memory of the key-value disk according to the address index, read the key-value pair, and thereby obtain the physical address block where the data block identified as H was previously stored, that is, the above-mentioned target value. The first logical address key and the target value form a key-value pair to be written, and are stored in the non-volatile memory of the key-value disk, that is, the first logical address specified by the host is associated with the physical address block previously stored in the data block identified as H, and the data block identified as H is no longer stored repeatedly. When the identification key is found in the record of the volatile memory, the on-disk processor of the key-value disk returns the information of the identification key found to the storage controller through the Store operation interface. After receiving the information, the storage controller no longer stores the data block identified as H, so as to avoid repeated storage of the data block.

[0063] Step S330: When the identification key is not found, the physical address block of the data block written by the storage controller is obtained as the target value, the identification key and the target value are used to form a first key-value pair, the first logical address key and the target value are used to form a second key-value pair, and the first key-value pair and the second key-value pair are determined to be the key-value pair to be written, and the key-value pair to be written is written into the non-volatile memory of the key-value disk. Specifically, when the identification key is not found, it means that the data block identified as H is not stored in the storage device, and the disk processor of the key-value disk returns a storage command to the storage controller through the Store operation interface to store the data block identified as H. After receiving the command, the storage controller stores the data block identified as H, obtains the corresponding physical address block, returns the physical address block through the Store operation interface, takes the physical address block as the target value, establishes a first key-value pair of the identification key and the target value, and a second key-value pair of the first logical address key and the target value, and writes the first key-value pair and the second key-value pair as the key-value pair to be written into the non-volatile memory of the key-value disk.

[0064] It should be noted that: in order to prevent hash conflicts and ensure that the corresponding data blocks are exactly the same when the identifiers are the same, the metadata I / O write request also includes: the data block to be written, when the identifier key is found, the data block is compared, if they are the same, the identifier key is determined to be found, otherwise, the information of the identifier conflict is returned to the storage controller, and the next metadata I / O request from the storage controller is waited. After receiving the information of the identifier conflict, the storage controller recalculates the identifier and issues the metadata I / O write request again.

[0065] In some embodiments, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes: step S130: metadata read operation, the second logical address key corresponds to the logical address block of the data block to be read by the host, such as Figure 1 As shown, step S130 specifically includes the following steps.

[0066] Step S131: searching for the address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk.

[0067] Step S132: Obtain a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key.

[0068] Step S133: Read the physical address block corresponding to the second logical address key in the nonvolatile memory based on the target storage address, and return the physical address block to the storage controller, so that the storage controller reads the data block to be read according to the physical address block and returns it to the host.

[0069] Specifically, the Exist operation interface of the key-value disk is called to send a metadata I / O read request. In this embodiment, the Exist operation interface can be implemented according to the above steps S131 to S133. Since the reading operation of the corresponding metadata is performed in the non-volatile memory of the key-value disk when reading the data block, the process of converting the cache data organization structure when the cache does not hit in the existing reading method is avoided, and the overall reading efficiency of the data block is improved.

[0070] In some embodiments, when the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes: step S140: metadata delete operation, the third logical address key corresponds to the logical address block of the data block to be deleted by the host, such as Figure 1 As shown, step S140 specifically includes the following steps.

[0071] Step S141: searching for the address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk.

[0072] Step S142: Obtain the target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key.

[0073] Step S143: deleting the key-value pair to be deleted from the non-volatile memory based on the target storage address.

[0074] Step S144: deleting the third logical address key and the corresponding address index from the record in the volatile memory of the key-value disk.

[0075] Specifically, the Delete operation interface of the key-value disk is called to send a metadata I / O deletion request. In this embodiment, the Delete operation interface can be implemented according to the above steps S141 to S144. Since it is necessary to access the metadata to locate the location of the data block to be deleted when deleting the data block, the access and deletion operations of the corresponding metadata are performed in the non-volatile memory of the key-value disk, which avoids the process of converting the cache data organization structure when the cache does not hit in the existing access method, and improves the overall deletion efficiency of the data block.

[0076] In some embodiments, the non-volatile memory includes: non-volatile memory and non-volatile flash memory (Flash). Based on this, in the metadata write operation, the step of writing the key-value pair to be written into the non-volatile memory of the key-value disk specifically includes: writing the key-value pair to be written into the non-volatile memory, and when the data size of the key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, transferring all key-value pairs corresponding to the storage block size in the non-volatile memory to the non-volatile flash memory, and modifying the address index of each transferred key-value pair in the record of the volatile memory. The non-volatile memory has a faster read and write speed, but the hardware cost is high. The read and write speed of the non-volatile flash memory is slower than that of the non-volatile memory, faster than the reading speed of the ordinary SSD disk, and the hardware cost is low. Therefore, in this embodiment, the non-volatile memory is used as a cache, and the non-volatile flash memory is used as a large-capacity metadata storage space, which reduces the hardware cost of the key-value disk while ensuring fast access to metadata.

[0077] Since non-volatile flash memory is stored based on a certain size of storage block as the minimum access unit, preferably, in order to facilitate the non-volatile memory to cache metadata while storing the storage block read out of the non-volatile flash memory in subsequent metadata read operations or delete operations, the capacity of the non-volatile memory is designed to be greater than twice the size of the storage block.

[0078] Furthermore, the capacity of the non-volatile memory is designed to be N times the size of the storage block, where N is an integer greater than or equal to 3. The step of writing the key-value pair to be written into the non-volatile memory of the key-value disk specifically includes: writing the key-value pair to be written into the non-volatile memory, and when the data size of the key-value pair in the non-volatile memory reaches N-1 times the size of the storage block, all key-value pairs corresponding to the size of n storage blocks are transferred to the non-volatile flash memory according to the preset transfer strategy, wherein the preset transfer strategy can be the nK (the size of K key-value pairs is the size of one storage block) key-value pairs with the least number of recent accesses. n=1,2,…,N-1. The size of n is set according to actual conditions, for example: n=1, at this time, there are always key-value pairs with the capacity of N-2 storage blocks in the non-volatile memory, which improves the hit probability of the key-value pair in the non-volatile memory and the access speed of the key-value pair, but increases the number of key-value pairs transferred from the non-volatile memory to the non-volatile flash memory. In this embodiment, n can be set to ensure that there are more key-value pairs in the non-volatile memory, improve the access speed of the key-value pairs, and minimize the number of times the key-value pairs are transferred from the non-volatile memory to the non-volatile flash memory, thereby reducing additional overhead. n is selected according to actual business conditions to achieve a balance between the access speed of the key-value pairs and the reduction of the overhead of the number of transfers. For example: in a scenario with more metadata writing business, n is set to a larger value, even n=N-1; in a scenario with more metadata reading business, n is set to a smaller value, even n=1.

[0079] For a non-volatile memory including: a non-volatile memory and a non-volatile flash memory (Flash), in some embodiments, when the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, such as Figure 4 The steps of the metadata reading operation include the following steps.

[0080] Step S410: searching for the address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk.

[0081] Step S420: Obtain a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key.

[0082] Step S430: when the target storage address is the address of the non-volatile memory, read the physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller.

[0083] Step S440: When the target storage address is the address of the non-volatile flash memory, read the target storage block where the target storage address is located from the non-volatile flash memory, then read the physical address block corresponding to the second logical address key from the target storage block, and return the physical address block to the storage controller. Specifically, read the target storage block from the non-volatile flash memory to the non-volatile memory, and after returning the physical address block to the storage controller, release the space of the target storage block in the non-volatile memory; or, release the space of the target storage block in the non-volatile flash memory, transfer it according to the above-mentioned preset transfer strategy, and modify the address index of each transferred key-value pair in the record of the volatile memory (here, transfer refers to transfer from the non-volatile flash memory to the non-volatile memory, and transfer from the non-volatile memory to the non-volatile flash memory).

[0084] For a non-volatile memory including: a non-volatile memory and a non-volatile flash memory (Flash), in some embodiments, when the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, such as Figure 5 The steps of the metadata deletion operation include the following steps.

[0085] Step S510: searching for the address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk.

[0086] Step S520: Obtain the target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key.

[0087] Step S530: When the target storage address is the address of the non-volatile memory, delete the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory, and delete the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk.

[0088] Step S540: When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory to the non-volatile memory (i.e., the data of the target storage block is read into the non-volatile memory), the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk. If the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory (or transferred according to the above-mentioned preset transfer strategy), and the address index of each transferred key-value pair is modified in the record of the volatile memory (here, transfer refers to transfer from the non-volatile flash memory to the non-volatile memory, and transfer from the non-volatile memory to the non-volatile flash memory).

[0089] In some embodiments, the metadata management method based on the online deduplication function also includes: when the key-value disk is started after power failure, traversing all key-value pairs in the non-volatile memory, rebuilding the address indexes of all key-value pairs, and storing the reconstructed address indexes and corresponding keys in the volatile memory. Among them, in the structure in which the non-volatile memory includes: non-volatile memory and non-volatile flash memory (Flash), both the non-volatile memory and the non-volatile flash memory need to be traversed. In this embodiment, by traversing the key-value pairs in the non-volatile memory of the key-value disk, the address index of the key-value pair and the record of the corresponding key can be quickly reconstructed.

[0090] The metadata management device based on the online deduplication function provided by the present invention is described below. The metadata management device based on the online deduplication function described below and the metadata management method based on the online deduplication function described above can be referred to each other.

[0091] The metadata management device based on online deduplication function of the embodiment of the present invention is as follows: Figure 6 As shown, it includes the following modules.

[0092] The I / O request receiving module 610 is used to receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of a data block as a logical address key, and a physical address block of a data block corresponding to each of the identifier and the logical address block as a value.

[0093] The metadata writing module 620 is used to, when the metadata I / O request is a write request, include an identification key and a first logical address key in the metadata, determine a key-value pair to be written based on the identification key and the first logical address key, write the key-value pair to be written into the non-volatile memory of the key-value disk, and record the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0094] In the metadata management device based on the online deduplication function of this embodiment, since the key-value pairs of the metadata and the corresponding address indexes are stored in the key-value disk during the metadata write operation, when the storage controller executes the online deduplication function, the metadata I / O request is sent to the key-value disk, and the key-value disk maintains and manages the metadata in the form of key-value pairs. The storage controller of the storage device no longer needs to implement processes such as metadata mirroring, power-off protection, and cache data organization structure conversion, thereby reducing the complexity of the metadata management process design on the storage controller of the storage device and reducing the consumption of CPU and memory resources in the storage device.

[0095] Optionally, the metadata writing module 620 includes the following modules.

[0096] The identification key search module is used to search for the identification key in the record of the volatile memory of the key-value disk.

[0097] A key-value pair writing module is used to, when the identification key is found, determine that the value corresponding to the identification key is a target value, form the key-value pair to be written with the first logical address key and the target value, and write the key-value pair to be written into the non-volatile memory of the key-value disk; when the identification key is not found, obtain the physical address block of the storage controller to write the data block as the target value, form a first key-value pair with the identification key and the target value, form a second key-value pair with the first logical address key and the target value, determine that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and write the key-value pairs to be written into the non-volatile memory of the key-value disk.

[0098] Optionally, in the case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the metadata management device based on the online deduplication function also includes: a metadata reading module 630, used to search for an address index corresponding to the second logical address key in the record of the volatile memory of the key-value disk; obtain a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; read a physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller.

[0099] Optionally, in the case that the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the metadata management device based on the online deduplication function also includes: a metadata deletion module 640, which is used to search for an address index corresponding to the third logical address key in the record of the volatile memory of the key-value disk; obtain a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; delete the key-value pair to be deleted from the non-volatile memory based on the target storage address; and delete the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk.

[0100] Optionally, the non-volatile memory includes: a non-volatile memory and a non-volatile flash memory; the metadata writing module 620 is specifically used to write the key-value pairs to be written into the non-volatile memory, and when the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

[0101] Optionally, in the case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the metadata management device based on the online deduplication function also includes: a metadata reading module 630, used to search for an address index corresponding to the second logical address key in the record of the volatile memory of the key-value disk; obtain a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; in the case where the target storage address is the address of the non-volatile memory, read a physical address block corresponding to the second logical address key in the non-volatile memory based on the target storage address, and return the physical address block to the storage controller; in the case where the target storage address is the address of the non-volatile flash memory, read the target storage block where the target storage address is located from the non-volatile flash memory, then read the physical address block corresponding to the second logical address key from the target storage block, and return the physical address block to the storage controller.

[0102] Optionally, in the case where the metadata I / O request is a deletion request, the metadata I / O request includes a third logical address key in the metadata, and the metadata management device based on the online deduplication function further includes: a metadata deletion module 640, which is used to search for an address index corresponding to the third logical address key in the record of the volatile memory of the key-value disk; obtain a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; and in the case where the target storage address is the address of the non-volatile memory, delete the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory, and delete the third logical address key and the pair in the record of the volatile memory of the key-value disk. corresponding address index; when the target storage address is the address of the non-volatile flash memory, read the target storage block where the target storage address is located from the non-volatile flash memory to the non-volatile memory, release the storage space corresponding to the target storage block in the non-volatile flash memory, delete the key-value pair to be deleted, and delete the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk; if the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, transfer all key-value pairs corresponding to the storage block size in the non-volatile memory to the non-volatile flash memory, and modify the address index of each transferred key-value pair in the record of the volatile memory.

[0103] Optionally, the metadata management device based on online deduplication function also includes: an address index reconstruction module, which is used to traverse all key-value pairs in the non-volatile memory when the key-value disk is started after power failure, rebuild the address indexes of all key-value pairs, and store the reconstructed address indexes and corresponding keys in the volatile memory.

[0104] The present invention also provides a key-value disk, comprising: an in-disk processor, a volatile memory and a non-volatile memory, wherein the in-disk processor is connected to the volatile memory and the non-volatile memory.

[0105] The non-volatile memory is used to store key-value pairs related to metadata, and the key-value pairs include: a key-value pair with an identifier of a data block as an identifier key and a physical address block of the data block as a value, and a key-value pair with a logical address block of the data block as a logical address key and a physical address block of the data block as a value.

[0106] The volatile memory is used to record the key of each key-value pair and the address index of the storage address of each key-value pair in the non-volatile memory.

[0107] The processor in the disk stores a computer program for implementing the metadata management method based on the online deduplication function as described in the above embodiments when executing the computer program.

[0108] Furthermore, the non-volatile memory includes: a non-volatile memory and a non-volatile flash memory.

[0109] The key-value disk is used to implement the key-value disk of the metadata management method based on the online deduplication function, and the key-value disk also has corresponding technical effects.

[0110] The present invention also provides a storage device, comprising: a storage controller and the above-mentioned key-value disk, wherein the storage controller is connected to the on-disk processor and is used to send a metadata I / O request to the on-disk processor, and the key-value disk is a KV SSD. Figure 2 , KV SSD is deployed in the storage device, and KV SSD implements the metadata management method based on the online deduplication function.

[0111] The storage device includes the key-value disk for implementing the metadata management method based on the online deduplication function, and the storage device also has the corresponding technical effect.

[0112] based on Figure 2 The process of implementing online deduplication when writing data on a storage device is as follows: Figure 7 As shown, the following steps are included.

[0113] Step S710: a storage controller of a storage device receives a data block to be written and a logic address block L. The data block to be written and the logic address block L are sent by a host through an I / O write request.

[0114] Step S720: the storage controller calculates the identifier H (hash value) of the data block to be written as the identifier of the data block to be written.

[0115] Step S730: the storage controller calls the interface (Store operation interface) of the KV SSD disk to perform fingerprint search.

[0116] Step S740: The processor in the KV SSD disk searches for a corresponding identification key based on the identification H. If a corresponding identification key is found, it means that the data block to be written has been stored in the storage device, and the HP key-value pair has been stored in the non-volatile memory of the KV SSD disk, then execute step S770, otherwise execute step S750.

[0117] Step S750: The data block to be written is written to the disk. When the identification key is not found, it means that the data block to be written identified as H is not stored in the storage device. The on-disk processor of the key-value disk will return a storage command to the storage controller through the Store operation interface to store the data block to be written identified as H. After receiving the command, the storage controller will store the data block to be written identified as H and obtain the corresponding physical address block P.

[0118] Step S760: Acquire the physical address block P, and write the HP key-value pair into the KV SSD disk. Specifically, write the HP key-value pair into the non-volatile memory of the KV SSD disk.

[0119] Step S770: Write the LP key-value pair to the KV SSD disk. Specifically, obtain the address index of the key-value pair corresponding to the identification key from the record of the volatile memory of the KV SSD disk, find the location where the key-value pair is stored in the non-volatile memory of the key-value disk according to the address index, read the key-value pair, and thereby obtain the physical address block P stored before the data block to be written marked as H, and write the LP key-value pair to the non-volatile memory of the KV SSD disk.

[0120] Figure 8 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the metadata management method based on the online deduplication function, and the method includes the following steps.

[0121] A metadata I / O request is received from a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0122] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into a non-volatile memory of a key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0123] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0124] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the metadata management method based on the online deduplication function provided by the above methods, and the method includes the following steps.

[0125] A metadata I / O request is received from a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0126] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into a non-volatile memory of a key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0127] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for metadata management based on online deduplication function provided by the above methods is implemented, and the method includes the following steps.

[0128] A metadata I / O request is received from a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value.

[0129] In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into a non-volatile memory of a key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

[0130] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0131] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0132] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A metadata management method based on online deduplication function, characterized in that: include: Receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value; In the case where the metadata I / O request is a write request, the metadata I / O request includes an identification key and a first logical address key in the metadata, determines a key-value pair to be written based on the identification key and the first logical address key, writes the key-value pair to be written into a non-volatile memory of a key-value disk, and records the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

2. The metadata management method based on online deduplication function according to claim 1, characterized in that: Based on the identification key and the first logical address key, determining a key-value pair to be written, and writing the key-value pair to be written into a non-volatile memory of the key-value disk, comprising: searching for the identification key in a record in a volatile memory of the key-value disk; In the case where the identification key is found, determining that the value corresponding to the identification key is a target value, forming the key-value pair to be written with the first logical address key and the target value, and writing the key-value pair to be written into the non-volatile memory of the key-value disk; If the identification key is not found, the physical address block of the data block written by the storage controller is obtained as the target value, a first key-value pair is formed with the identification key and the target value, a second key-value pair is formed with the first logical address key and the target value, it is determined that the first key-value pair and the second key-value pair are both the key-value pairs to be written, and the key-value pairs to be written are written into the non-volatile memory of the key-value disk.

3. The metadata management method based on online deduplication function according to claim 1, characterized in that: In the case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk; Acquire a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; A physical address block corresponding to a second logical address key in a nonvolatile memory is read based on the target storage address, and the physical address block is returned to the storage controller.

4. The metadata management method based on online deduplication function according to claim 1, characterized in that: In the case where the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk; Acquire a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; Deleting the key-value pair to be deleted from the non-volatile memory based on the target storage address; The third logical address key and the corresponding address index are deleted from the record of the volatile memory of the key-value disk.

5. The metadata management method based on online deduplication function according to claim 1, characterized in that: The non-volatile memory includes: a non-volatile memory and a non-volatile flash memory; writing the key-value pair to be written into the non-volatile memory of the key-value disk includes: The key-value pairs to be written are written into the non-volatile memory. When the data size of the key-value pairs in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

6. The metadata management method based on online deduplication function according to claim 5, characterized in that: In the case where the metadata I / O request is a read request, the metadata I / O request includes a second logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the second logical address key in the records of the volatile memory of the key-value disk; Acquire a target storage address of the key-value pair to be read based on the address index corresponding to the second logical address key; In a case where the target storage address is the address of the non-volatile memory, reading a physical address block corresponding to a second logical address key in the non-volatile memory based on the target storage address, and returning the physical address block to the storage controller; When the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory, and then the physical address block corresponding to the second logical address key is read from the target storage block, and the physical address block is returned to the storage controller.

7. The metadata management method based on online deduplication function according to claim 5, characterized in that: In the case where the metadata I / O request is a delete request, the metadata I / O request includes a third logical address key in the metadata, and the method further includes: Searching for an address index corresponding to the third logical address key in the records of the volatile memory of the key-value disk; Acquire a target storage address of the key-value pair to be deleted based on the address index corresponding to the third logical address key; In the case where the target storage address is the address of the non-volatile memory, deleting the key-value pair to be deleted corresponding to the target storage address in the non-volatile memory, and deleting the third logical address key and the corresponding address index in the record of the volatile memory of the key-value disk; In the case where the target storage address is the address of the non-volatile flash memory, the target storage block where the target storage address is located is read from the non-volatile flash memory to the non-volatile memory, the storage space corresponding to the target storage block in the non-volatile flash memory is released, the key-value pair to be deleted is deleted, and the third logical address key and the corresponding address index are deleted in the record of the volatile memory of the key-value disk; if the data size of the current key-value pair in the non-volatile memory reaches the storage block size of the non-volatile flash memory, all key-value pairs corresponding to the storage block size in the non-volatile memory are transferred to the non-volatile flash memory, and the address index of each transferred key-value pair is modified in the record of the volatile memory.

8. The metadata management method based on online deduplication function according to any one of claims 1 to 7, characterized in that: Also includes: When the key-value disk is started after power failure, all key-value pairs in the non-volatile memory are traversed, address indexes of all key-value pairs are rebuilt, and the rebuilt address indexes and corresponding keys are stored in the volatile memory.

9. A metadata management device based on online deduplication function, characterized in that: include: An I / O request receiving module is used to receive a metadata I / O request sent by a storage controller in a storage device, wherein the metadata includes an identifier of a data block as an identifier key, a logical address block of the data block as a logical address key, and a physical address block of the data block corresponding to each of the identifier and the logical address block as a value; A metadata writing module is used to, when the metadata I / O request is a write request, determine a key-value pair to be written based on the identification key and the first logical address key, write the key-value pair to be written into the non-volatile memory of the key-value disk, and record the key in the key-value pair to be written and the address index of the storage address of the key-value pair to be written in the non-volatile memory in the volatile memory of the key-value disk.

10. A key-value disk, characterized in that: include: An in-disk processor, a volatile memory and a non-volatile memory, wherein the in-disk processor is connected to the volatile memory and the non-volatile memory; The non-volatile memory is used to store key-value pairs related to metadata, the key-value pairs including: a key-value pair with an identifier of a data block as an identifier key and a physical address block of the data block as a value, and a key-value pair with a logical address block of the data block as a logical address key and a physical address block of the data block as a value; The volatile memory is used to record the key of each key-value pair and the address index of the storage address of each key-value pair in the non-volatile memory; The processor in the disk stores a computer program for implementing the metadata management method based on the online deduplication function as claimed in any one of claims 1 to 8 when executing the computer program.

11. A storage device, characterized in that: include: A storage controller and the key-value disk as claimed in claim 10, wherein the storage controller is connected to the on-disk processor and is used to send metadata I / O requests to the on-disk processor.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the metadata management method based on the online deduplication function is implemented as described in any one of claims 1 to 8.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the metadata management method based on the online deduplication function is implemented as described in any one of claims 1 to 8.

Images