A data deduplication method, device, equipment and readable storage medium

Abstract

The application discloses a data deduplication method, device and equipment and a readable storage medium. The method comprises the following steps: obtaining a data block to be written and a target volume logical address, and calculating a target hash value of the data block; in an HP metadata cache, querying an HP record based on the target hash value; if the HP record of the data block is not found, writing the data block into a target storage pool address, inserting the HP record of the data block into the HP metadata cache and inserting an LP record of the data block into LP metadata; if the HP record of the data block is found, inserting a PL record of the data block into PL metadata and inserting the LP record of the data block into the LP metadata, and updating the HP record of the data block in the HP metadata cache. Technical effects: in the case that the data is not repeatedly written, the PL record is not inserted, the write amplification problem of metadata processing can be effectively solved, the metadata processing overhead can be reduced, and the performance is improved.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to a data deduplication method, apparatus, device, and readable storage medium. Background Technology

[0002] SAN (Storage Area Network) all-flash storage systems provide block storage services through network and protocols. The system manages multiple SSDs (Solid State Disks), arranging them into RAID (Redundant Arrays of Independent Disks) and storage pools, and allocating virtual space as LUNs (Logical Unit Numbers) for the host. Hereinafter, the SAN storage system will be referred to as the storage system. It performs secondary management of hard drives and RAID space, using metadata to map data from volume logical blocks (LBAs) to storage pool block addresses (PBAs), enabling large-block aggregation writes across RAID stripes to improve performance and reduce write amplification.

[0003] To further improve storage space utilization, data deduplication can be performed. Currently, LBA-to-PBA mappings point duplicate volume logical blocks to the same physical block. This requires recording how many LBA mappings and references exist for the same PBA; this recorded information is the PL metadata. Additionally, it involves calculating and recording the mapping information from data block fingerprints (i.e., hashes) to PBAs; this mapping information is the HP metadata. However, real-time recording of PL and HP metadata involves memory caching and reading / writing to the backend disk, resulting in high metadata processing overhead and a significant performance impact.

[0004] In conclusion, how to effectively solve problems such as data deduplication is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a data deduplication method, apparatus, device, and readable storage medium that can record PLs only for duplicate data when writing data, and not record PLs for non-duplicate data, thereby reducing the metadata that needs to be recorded in real time when writing data, reducing metadata processing overhead, and improving system performance.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution: A data deduplication method includes: Obtain the data block to be written and the target volume logical address, and calculate the target hash value of the data block; In the HP metadata cache, query the HP record of the data block based on the target hash value; If the HP record of the data block is not found, the data block is written to the newly allocated target storage pool address, and the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata cache, in combination with the target volume logical address and the target storage pool address. If the HP record of the data block is found, then the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, and the HP record of the data block in the HP metadata cache is updated. The HP record is a mapping information between the hash value of a data block and the address of the storage pool. The LP record is the mapping information between the volume logical address and the storage pool address of the data block; The PL record contains mapping information between the storage pool address and the volume logical address of the data block.

[0007] Preferably, it further includes: If the cache space is insufficient, the target HP record will be evicted from the HP metadata cache; The target log block that is closest to the hash value in the target HP record is determined from the HP metadata log area; wherein, the HP metadata log area includes several log blocks divided according to different hash value ranges; Write the log entry corresponding to the target HP record into the target log block; wherein the log entry includes a hash value, a volume logical address, and a storage pool address.

[0008] Preferably, it further includes: Log entries are read from the HP metadata log area in the order of newest to oldest, using a single log block as the reading unit. The read log entries are filtered to obtain the valid log entries; Update the HP metadata cache using the valid log entries.

[0009] Preferably, the read log entries are filtered to obtain valid log entries, including: In the LP metadata, LP record queries are performed based on the hash value in the read log item; If the target LP record is not found, the log entry being read is determined to be an invalid log entry. If the target LP record is found, the target LP record and the log item are matched and verified. Log entries that fail the matching verification are identified as invalid log entries; Log entries that pass the matching verification are identified as valid log entries.

[0010] Preferably, updating the HP metadata cache using the valid log entries includes: HP record lookup is performed from the HP metadata cache based on the hash in the valid log entry; If a corresponding HP record is found, the valid log entries are ignored. If no corresponding HP record is found, an HP record matching the valid log entry is added to the HP metadata cache.

[0011] Preferably, the HP record of the data block records the hash value, storage pool address, volume logical address, and data deduplication count of the data block. Accordingly, if the HP record of the data block is found, the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, based on the target volume logical address. The HP record of the data block in the HP metadata cache is then updated, including: Determine whether the data duplication count in the HP record of the data block is 1; If so, the first volume logical address and the first storage pool address are read from the HP record of the data block; The address of the first storage pool is queried from the LP metadata based on the logical address of the first volume; If the first storage pool address is not found, after deleting the HP record of the data block in the HP metadata cache, the data block is written to the newly allocated target storage pool address; combining the target volume logical address and the target storage pool address, the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata. If the address of the first storage pool is found, then the PL record of the last time the data block was written is inserted into the PL metadata in combination with the logical address of the first volume and the address of the first storage pool. The PL record of the current time the data block is written is inserted into the PL metadata in combination with the logical address of the target volume. The LP record of the current time the data block is written is inserted into the LP metadata. The data duplication count in the HP record of the data block in the HP metadata cache is incremented by one. If not, then, based on the target volume logical address, insert the PL record for this write to the data block into the PL metadata and the LP record for this write to the data block into the LP metadata, and increment the data duplication count in the HP record of the data block in the HP metadata cache by one.

[0012] Preferably, before inserting LP data into the LP metadata, the method further includes: Determine whether a corresponding old PL record exists for the old LP record corresponding to the logical address of the target volume; If so, delete the old PL record.

[0013] A data deduplication device, comprising: The hash calculation module is used to obtain the data block to be written and the target volume logical address, and to calculate the target hash value of the data block; The data deduplication module is used to query the HP record of the data block in the HP metadata cache based on the target hash value; The metadata omission record module is used to write the data block into the newly allocated target storage pool address if no HP record of the data block is found, and to insert the HP record of the data block into the HP metadata cache and the LP record of the data block into the LP metadata cache by combining the target volume logical address and the target storage pool address. The metadata comprehensive recording module is used to, if the HP record of the data block is found, insert the PL record of the data block into the PL metadata and the LP record of the data block into the LP metadata, and update the HP record of the data block in the HP metadata cache, in combination with the target volume logical address. The HP record is a mapping information between the hash value of a data block and the address of the storage pool. The LP record is the mapping information between the volume logical address and the storage pool address of the data block; The PL record contains mapping information between the storage pool address and the volume logical address of the data block.

[0014] An electronic device, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the above-described data deduplication method when executing the computer program.

[0015] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described data deduplication method.

[0016] Applying the method provided in this application embodiment, the data block to be written and the target volume logical address are obtained, and the target hash value of the data block is calculated. In the HP metadata cache, the HP record of the data block is queried based on the target hash value. If the HP record of the data block is not found, the data block is written to the newly allocated target storage pool address, and the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata, combined with the target volume logical address and the target storage pool address. If the HP record of the data block is found, the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, and the HP record of the data block in the HP metadata cache is updated. Herein, the HP record is the mapping information between the hash value of the data block and the storage pool address; the LP record is the mapping information between the volume logical address of the data block and the storage pool address; and the PL record is the mapping information between the storage pool address and the volume logical address of the data block.

[0017] Technical Effects: In the process of data deduplication based on metadata, HP metadata is cached. Therefore, when a data block needs to be written, it can first be determined whether the data block has been written repeatedly based on the HP metadata. If it is determined to be a repeated write, the data block is not written; instead, LP and PL records are inserted, and the HP record is updated but not written to disk in real time. If it is not a repeated write, the data block is written, and only LP and HP records are inserted, without inserting the LP record. In other words, by caching HP metadata and determining whether duplicate data is written based on the cached metadata, inserting LP and PL records only when duplicate data is written, and not inserting PL records when no duplicate data is written, the write amplification problem in metadata data processing can be effectively solved, reducing metadata processing overhead and improving performance.

[0018] Accordingly, embodiments of this application also provide a data deduplication apparatus, device, and readable storage medium corresponding to the above-described data deduplication method, which have the aforementioned technical effects, and will not be elaborated further here. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the implementation of a data deduplication method in an embodiment of this application. Figure 2 This is a schematic diagram illustrating an implementation of a data deduplication method in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a data deduplication device according to an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application; Figure 5 This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] This application relates to methods for processing data and metadata in all-flash SAN storage systems. SAN storage systems provide host logical block devices, or logical volumes, through a SAN network. The hardware of a SAN storage system includes computing subsystems such as CPUs and memory, front-end and back-end I / O cards, hard drive enclosures, and hard drives. The software includes operating systems, storage management, and data processing software systems. These software systems typically include modules such as front-end and back-end I / O card drivers, targets, caches, storage pools, and RAID, each corresponding to its respective I / O stack layer. This invention primarily relates to metadata and data deduplication processing at the storage pool layer.

[0023] The system allocates a certain amount of memory space and capacity for HP metadata caching. When the memory is large enough, the HP metadata cache can cache all HP metadata. HP metadata is indexed by data block fingerprint Hash and includes Hash, PBA, LBA, and HitCount. HitCount represents the number of times the data fingerprint is repeated when processing write data. The index structure can adopt hash table, base tree, or other structures.

[0024] For details, please refer to Figure 1 and Figure 2 This method can be applied to SAN storage systems and includes the following steps: S101. Obtain the data block to be written and the logical address of the target volume, and calculate the target hash value of the data block.

[0025] Upon receiving the data block to be written and the target volume logical address, the target hash value of the data block is first calculated.

[0026] In this embodiment, no limitations are imposed on the data content of the data block to be written, the specific address of the target volume logical address, or the hash algorithm used to calculate the hash of the data block.

[0027] It should be noted that the target volume logical address refers to the volume logical address where the data block is to be written, and the target hash value is the hash value of the data block. The word "target" is used here to distinguish it from the volume logical address and hash value of other data blocks, and has no other limiting meaning.

[0028] S102. In the HP metadata cache, query the HP record of the data block based on the target hash value.

[0029] HP metadata includes metadata that records the mapping information between the hash value of a data block and the storage pool address.

[0030] In this embodiment, to reduce metadata writing, HP metadata is cached in memory. Furthermore, the HP records within this HP metadata contain HP records of data blocks that have already been written to, recording information such as the hash value, PBA, and LBA of the written data blocks. Therefore, once the target hash value of the data block to be written is determined, the HP record of the data block can be directly queried from the HP metadata cache based on the target hash value.

[0031] Different follow-up processing operations can be performed based on different query results.

[0032] HP metadata cache can be saved to persistent media when the system is shut down and loaded and restored to memory upon restart.

[0033] S103. If no HP record for the data block is found, the data block is written to the newly allocated target storage pool address. Combined with the target volume logical address and the target storage pool address, the HP record for the data block is inserted into the HP metadata cache and the LP record for the data block is inserted into the LP metadata cache.

[0034] If no HP record for the data block to be written is found in the HP metadata cache, it indicates that this data block is being written for the first time and there is no duplicate data. Therefore, a target storage pool address needs to be assigned to this data block. The target storage pool address is the storage pool address of the data block, and its purpose is to distinguish it from the storage pool addresses of other data blocks.

[0035] After determining the target storage pool address, the data block is written to that address, and an HP record for the data block is inserted into the HP metadata cache, while an LP record for the data block is inserted into the LP metadata cache. In other words, the metadata operations required for the initial data block write include: 1. Insert the LP record of the data block into the LP metadata; 2. Insert the HP record for this data block into the HP metadata cache.

[0036] For example, if the hash value of the data block is H1, the target logical volume address is L1, and the target storage pool address is P1, then the LP record is L1P1, and the HP record is indexed by H1, including H1, L1, P1, and HitCount=1. Here, HitCount is the data duplication count, and a value of 1 indicates the first write.

[0037] It is important to note that when no duplicate data blocks are found, the PL record of the data block is not inserted into the PL metadata during metadata insertion. This reduces the amount of metadata processed during data writing and effectively reduces metadata processing overhead.

[0038] S104. If the HP record of the data block is found, then insert the PL record of the data block into the PL metadata and the LP record of the data block into the LP metadata, and update the HP record of the data block in the HP metadata cache, based on the target volume logical address.

[0039] Among them, HP records are mapping information between the hash value of the data block and the storage pool address; The LP record contains mapping information between the volume logical address and the storage pool address of the data block; PL records the mapping information between the storage pool address and the volume logical address of the data block.

[0040] HP records can be specifically indexed by hash and include mapping information for hash, LBA, PBA, and data duplication count; LP records can be specifically indexed by LBA and include mapping information for LBA and PBA; PL records can be specifically indexed by PBA and include mapping information for PBA and LBA.

[0041] If the HP record of the data block can be found, it indicates that the data block is not being written for the first time and data deduplication is required. That is, the CNC block will no longer be written to the storage pool. Instead, a mapping relationship will be established between the logical volume address of this write, i.e. the target logical volume address, and the storage pool address of the already written data block, and the mapping information will be recorded.

[0042] Specifically, the metadata operations performed at this time include: 1. Insert the PL record of the data block into the PL metadata; 2. Insert the LP record of the data block into the LP metadata; 3. Update the HP records of the data blocks in the HP metadata cache.

[0043] For example, if the hash value of the data block is H1, the target logical volume address is L1, and the target storage pool address is P1, then the newly inserted LP record is L1P1, and the PL record includes the LP record P1L1 written in this data block; the newly inserted HP record is indexed by H1 and includes H1, L1, P1, and HitCount=2. Here, HitCount is the data duplication count, and a value of 2 indicates a second write.

[0044] In one specific embodiment of this application, the number of times data is repeated can also be recorded, and data deduplication processing can be performed based on different numbers of data repetitions. Specifically, the HP record of a data block records the hash value of the data block, the storage pool address, the volume logical address, and the data repetition count; Accordingly, if the HP record for the data block is found, the PL record for the data block is inserted into the PL metadata and the LP record for the data block is inserted into the LP metadata, based on the target volume logical address. The HP record for the data block in the HP metadata cache is then updated, including: Determine if the data duplication count in the HP record of the data block is 1; If so, read the first volume logical address and the first storage pool address from the HP record of the data block; Query the address of the first storage pool based on the logical address of the first volume in the LP metadata; If the first storage pool address is not found, after deleting the HP record of the data block in the HP metadata cache, the data block is written to the newly allocated target storage pool address; combining the target volume logical address and the target storage pool address, the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata. If the address of the first storage pool is found, the PL record of the last written data block is inserted into the PL metadata by combining the logical address of the first volume and the address of the first storage pool. The PL record of the current written data block is inserted into the PL metadata by combining the logical address of the target volume. The LP record of the current written data block is inserted into the LP metadata. The data duplication count in the HP record of the data block in the HP metadata cache is incremented by one. If not, then based on the target volume logical address, insert the PL record of this data block in the PL metadata and the LP record of this data block in the LP metadata, and increment the data duplication count in the HP record of the data block in the HP metadata cache.

[0045] For ease of description, the above steps will be combined below.

[0046] If the HP record for the data block can be found, it indicates that this write is not the first time the data block has been written. By querying the data duplication count in the HP record, it can be determined how many times the data block has been written.

[0047] For example, when writing the data block for the first time, Hash, PBA, LBA, and HitCount are recorded in the HP metadata cache. Suppose Hash=H0, PBA=P0, LBA=L0, and HitCount=1, where HitCount is set to 1 to indicate the first write.

[0048] When writing a data block, if the data duplication count in the HP record corresponding to the data block is 1, it indicates that this is the second time the data block has been written. At this point, the validity of the HP record is first verified. Specifically, the logical address of the first volume and the address of the first storage pool can be read from the HP record corresponding to the data block. If the HP record corresponding to the data block in the HP metadata cache has Hash=H0, PBA=P0, LBA=L0, and HitCount=1, then the logical address of the first volume is L0, and the address of the first storage pool is P0. To verify the validity of the HP record, the corresponding P0 can be queried from the LP metadata based on L0. If P0 can be found, the HP record is valid; otherwise, the HP record is invalid.

[0049] If the HP record is determined to be invalid, it can be deleted, and the data write process can proceed as if it were the first time the data block was written. Specifically, a target storage pool address needs to be allocated for the data block. After determining the target storage pool address, the data block is written to that address, the HP record for the data block is inserted into the HP metadata cache, and the LP record for the data block is inserted into the LP metadata cache. In other words, the metadata operations required for this data block write include: 1. Insert the LP record of the data block into the LP metadata; 2. Insert the HP record for this data block into the HP metadata cache.

[0050] If the HP record is confirmed to be valid, there is no need to write the data block. Instead, a mapping relationship is established between the logical volume address being written (i.e., the target logical volume address) and the storage pool address of the already written data block, and the mapping information is recorded. Since this is the second time the data block has been written, and the PL record for this data block was not inserted during the first write, the metadata operations performed this time include: 1. Insert the PL record for this data block write and the PL record for the first write into the PL metadata; 2. Insert the LP record for this data block into the LP metadata; 3. Update the HP records of the data blocks in the HP metadata cache.

[0051] When the data duplication count in the HP record of a data block is greater than 1, it indicates that this data block has been written for the third or more times. In this case, there is no need to continue writing to this data block. Instead, a mapping should be established between the logical volume address of this data write and the storage pool address of the previous data block write. That is, a mapping relationship is established between the logical volume address of this write (i.e., the target logical volume address) and the storage pool address of the already written data block, and the mapping information is recorded.

[0052] Specifically, the metadata operations performed at this time include: 1. Insert the PL record for this data block into the PL metadata; 2. Insert the LP record for this data block into the LP metadata; 3. Update the HP records of the data blocks in the HP metadata cache.

[0053] In one specific embodiment of this application, before inserting LP data into the LP metadata, the method further includes: Determine if a corresponding old PL record exists for the old LP record corresponding to the logical address of the target volume. If so, delete the old PL record; If not, delete the old LP record.

[0054] In other words, before inserting LP data into the LP metadata, if the old LP corresponding to the target volume logical address has a corresponding old PL record, the old PL record is deleted. Alternatively, the old LP can also be deleted. If the old LP record has no corresponding old PL record, the old LP record is deleted. This allows for updating the content of data blocks at the same logical volume address.

[0055] For example, if the target logical volume address is L1, and the old LP corresponding to L1 has PL metadata such as P1L1, then P1L1 needs to be deleted.

[0056] For example, when data needs to be written, firstly, the hash fingerprint value of the data block is calculated according to the data block size (such as a pre-set block size of 4K or 8K) and alignment. Assuming that the volume logical block number and address (i.e., LBA) of the data block being written is L1, and the calculated data hash fingerprint value is H0, firstly, the HP metadata cache is checked to see if there is a corresponding HP record based on H0.

[0057] If no HP record is found, the data block is written to the newly allocated location, and an HP record is inserted into the HP metadata cache. The HP record includes Hash, PBA, LBA, and HitCount. Assume Hash=H0, PBA=P0, LBA=L0, and HitCount=1, where HitCount is set to 1 to indicate the first write. L0P0 is then inserted into the LP metadata.

[0058] If a record exists and HitCount=1, check if L0P0 is valid. If invalid, delete the record and process it according to the initial write procedure. Otherwise, it indicates that the data is duplicated for the first time. In this case, insert L1P0 into the LP metadata and P0L1 and P0L0 into the PL metadata. At the same time, increment the HitCount of the information corresponding to H0 in the HP metadata. If the old LP corresponding to L1 has PL metadata such as P1L1, then P1L1 also needs to be deleted.

[0059] If a record exists and HitCount > 1, it indicates that the data is duplicated, and this is not the first time it has been duplicated. In this case, L1P0 is inserted into the LP metadata, and P0L1 is inserted into the PL metadata. At the same time, the HitCount of the information corresponding to H0 in the HP metadata is incremented by 1. If the old LP corresponding to L1 has PL metadata such as P1L1, then P1L1 also needs to be deleted.

[0060] Applying the method provided in this application embodiment, the data block to be written and the target volume logical address are obtained, and the target hash value of the data block is calculated. In the HP metadata cache, the HP record of the data block is queried based on the target hash value. If the HP record of the data block is not found, the data block is written to the newly allocated target storage pool address, and the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata, combined with the target volume logical address and the target storage pool address. If the HP record of the data block is found, the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, and the HP record of the data block in the HP metadata cache is updated. Herein, the HP record is the mapping information between the hash value of the data block and the storage pool address; the LP record is the mapping information between the volume logical address of the data block and the storage pool address; and the PL record is the mapping information between the storage pool address and the volume logical address of the data block.

[0061] Technical Effects: In the process of data deduplication based on metadata, HP metadata is cached. Therefore, when a data block needs to be written, it can first be determined whether the data block has been written repeatedly based on the HP metadata. If it is determined to be a repeated write, the data block is not written; instead, LP and PL records are inserted, and the HP record is updated but not written to disk in real time. If it is not a repeated write, the data block is written, and only LP and HP records are inserted, without inserting the LP record. In other words, by caching HP metadata and determining whether duplicate data is written based on the cached metadata, inserting LP and PL records only when duplicate data is written, and not inserting PL records when no duplicate data is written, the write amplification problem in metadata data processing can be effectively solved, reducing metadata processing overhead and improving performance.

[0062] It should be noted that, based on the above embodiments, the embodiments of this application also provide corresponding improvement schemes. In the preferred / improved embodiments, the same or corresponding steps as in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other; however, these will not be elaborated upon in the preferred / improved embodiments herein.

[0063] In one specific embodiment of this application, considering insufficient cache space, some HP metadata can be evicted and recorded in the log, thereby ensuring that HP metadata is not lost and ensuring the effective operation of the system. The specific implementation process includes: If the cache space is insufficient, the target HP record will be evicted from the HP metadata cache; The target log block that is closest to the hash value in the target HP record is identified from the HP metadata log area; the HP metadata log area includes several log blocks divided according to different hash value ranges; Write the log entry corresponding to the target HP record into the target log block; the log entry includes the hash value, volume logical address, and storage pool address.

[0064] For ease of description, the above steps will be combined below.

[0065] Specifically, in practical applications, a certain amount of memory space can be allocated for caching HP metadata. When the used space of this memory space is insufficient, it can be determined that the cache space is insufficient.

[0066] When cache space is insufficient, the target HP record can be evicted from the HP metadata cache. Here, the target HP record refers to the HP metadata that needs to be evicted from the cache. The word "target" is only to distinguish it from other HP records that are not evicted, and has no other limiting meaning.

[0067] In one specific implementation, when evicting HP metadata, the target HP record is selected using either the Least Recently Used (LRU) algorithm or random load balancing. That is, during data writing, if the HP metadata cache is insufficient for inserting a new fingerprint value, some HP metadata needs to be evicted to release the cache. The eviction method can use the Least Recently Used (LRU) algorithm or a random load balancing principle based on the PBA space or fingerprint value (hash value). The Least Recently Used (LRU) record can be selected to reduce data duplication; alternatively, random selection can achieve load balancing. The required HP metadata (i.e., HP records) is then selected, and the evicted HP records are written to the HP metadata log area.

[0068] In this embodiment, metadata in the HP metadata log area can be read asynchronously to analyze data duplication rates and perform data deduplication, thereby improving the pool space deduplication coverage and data deduplication rate. To facilitate finding all log entries corresponding to the same data block, the HP metadata log area can be divided into multiple log blocks according to the fingerprint value range, making the hash values ​​of HP within the same log block similar. For example, the hash value is 32 bytes long (256 bits), the log area is divided into M blocks, and a fixed N bits, such as the high 16 bits, are selected from the hash value as an index to map to the M blocks. The high 16 bits of the hash value of the HP metadata recorded in each block are the same.

[0069] Therefore, after identifying the target HP record, the first step is to determine the target log block from the HP metadata log area that most closely matches the hash value in the target HP record. Then, the corresponding log entry for the target HP record is written into the target log block. This log entry includes the hash value, volume logical address, and storage pool address. Specifically, each log block can use a circular rolling recording method, and each log entry includes a hash, PBA, and LBA.

[0070] In one specific embodiment of this application, the HP metadata cache is updated based on logs. The specific implementation process includes: Log entries are read from the HP metadata log area in the order of newest to oldest, using individual log blocks as the reading unit. The read log entries are filtered to obtain the valid log entries; Update the HP metadata cache using valid log entries.

[0071] For ease of description, the above steps will be combined and explained in detail below.

[0072] When reading logs, individual log blocks can be used as the unit of reading, and log entries can be read from the HP metadata log area in order from newest to oldest. Log entries are stored in partitioned blocks based on hash value ranges; therefore, reading individual log blocks allows for efficient reading of all log entries corresponding to the same data block.

[0073] After reading the log entries, they need to be filtered to obtain valid log entries. Then, the valid log entries are used to update the HP metadata cache.

[0074] In one specific implementation, the read log entries are filtered to obtain valid log entries, including: In the LP metadata, LP record queries are performed based on the hash value in the read log item; If the target LP record is not found, the log entry being read is determined to be an invalid log entry. If the target LP record is found, then the target LP record and the log item are matched and verified. Log entries that fail the matching verification are identified as invalid log entries; Log entries that pass the matching verification are identified as valid log entries.

[0075] In other words, LP records can be queried in the LP metadata based on the hash value in the read log entry. If the corresponding target LP log cannot be found, the log entry is invalid. If the corresponding target LP log can be found, further verification is needed based on whether the target LP record matches the read log entry, i.e., whether their PBAs are consistent. If they are consistent, it indicates a match. If they match, the target log entry is a valid log entry; otherwise, it is an invalid log entry.

[0076] In one specific implementation, updating the HP metadata cache using valid log entries includes: HP record lookup is performed from the HP metadata cache based on the hash in the valid log entry. If a corresponding HP record is found, the valid log entries are ignored; If no corresponding HP record is found, add an HP record that matches the valid log entry to the HP metadata cache.

[0077] For ease of description, the above steps will be combined below.

[0078] Once valid log entries are selected, the system can query the HP metadata cache to see if a corresponding HP record exists. Specifically, the query is performed using the hash value from the valid log entries.

[0079] If the corresponding HP record can be found, the valid log entry can be ignored.

[0080] If the corresponding HP record cannot be found, an HP record matching the valid log entry can be added to the HP metadata cache. For example, insert HASH, PBA, and LBA based on the valid log entry and set HitCount=1.

[0081] In one specific implementation, if a corresponding HP record is found in the HP metadata cache, further data deduplication can be performed based on the HP record and valid log entries. The specific implementation process includes: HP records are retrieved by querying the hash in valid log entries. Based on the hash of the valid log entries, read the LP records to obtain several LP records; Determine whether the storage pool addresses in the read LP records are consistent; If there is a discrepancy, modify the LP record to ensure that the storage pool addresses of the multiple LP records are consistent. The corresponding LP metadata, PL metadata, and HP metadata caches should be modified.

[0082] For example, if the valid log entry is hash=H5, LBA=L4, PBA=3, and the read LP record includes L4P3 and L3P2, then L3P2 needs to be changed to L3P3, P2 needs to be recycled, and PL needs to be changed to P3L3 and P3L4 respectively. Correspondingly, the HP record corresponding to H5 in the HP metadata cache needs to be changed to 2 for data duplication.

[0083] Corresponding to the above method embodiments, this application also provides a data deduplication device, and the data deduplication device described below can be referred to in correspondence with the data deduplication method described above.

[0084] See Figure 3 As shown, the device includes the following modules: The hash calculation module 101 is used to obtain the data block to be written and the logical address of the target volume, and to calculate the target hash value of the data block; Data deduplication module 102 is used to query HP records of data blocks in the HP metadata cache based on the target hash value; The metadata omission record module 103 is used to write the data block to the newly allocated target storage pool address if no HP record for the data block is found, and to insert the HP record of the data block into the HP metadata cache and the LP record of the data block into the LP metadata cache by combining the target volume logical address and the target storage pool address. The metadata comprehensive recording module 104 is used to, if the HP record of the data block is found, insert the PL record of the data block into the PL metadata and the LP record of the data block into the LP metadata, and update the HP record of the data block in the HP metadata cache, in combination with the target volume logical address. Among them, HP records are mapping information between the hash value of the data block and the storage pool address; The LP record contains mapping information between the volume logical address and the storage pool address of the data block; PL records the mapping information between the storage pool address and the volume logical address of the data block.

[0085] Using the apparatus provided in this application embodiment, the data block to be written and the target volume logical address are obtained, and the target hash value of the data block is calculated. In the HP metadata cache, the HP record of the data block is queried based on the target hash value. If the HP record of the data block is not found, the data block is written to the newly allocated target storage pool address, and combined with the target volume logical address and the target storage pool address, the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata. If the HP record of the data block is found, combined with the target volume logical address, the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, updating the HP record of the data block in the HP metadata cache. Here, the HP record is the mapping information between the hash value of the data block and the storage pool address; the LP record is the mapping information between the volume logical address of the data block and the storage pool address; and the PL record is the mapping information between the storage pool address and the volume logical address of the data block.

[0086] Technical Effects: In the process of data deduplication based on metadata, HP metadata is cached. Therefore, when a data block needs to be written, it can first be determined whether the data block has been written repeatedly based on the HP metadata. If it is determined to be a repeated write, the data block is not written; instead, LP and PL records are inserted, and the HP record is updated but not written to disk in real time. If it is not a repeated write, the data block is written, and only LP and HP records are inserted, without inserting the LP record. In other words, by caching HP metadata and determining whether duplicate data writes exist based on the cached metadata, inserting LP and PL records only when duplicate writes occur, and not inserting PL records when no duplicate writes occur, the write amplification problem in metadata processing can be effectively solved, reducing metadata processing overhead and improving performance.

[0087] In one specific embodiment of this application, a log management module is also included, which is used to evict the target HP record from the HP metadata cache if the cache space is insufficient; The target log block that is closest to the hash value in the target HP record is identified from the HP metadata log area; the HP metadata log area includes several log blocks divided according to different hash value ranges; Write the log entry corresponding to the target HP record into the target log block; the log entry includes the hash value, volume logical address, and storage pool address.

[0088] In one specific embodiment of this application, it further includes: an HP metadata update module, used to read log items from the HP metadata log area in the order from latest to oldest, with a single log block as the reading unit; The read log entries are filtered to obtain the valid log entries; Update the HP metadata cache using valid log entries.

[0089] In one specific embodiment of this application, the HP metadata update module is specifically used to query LP records in the LP metadata based on the hash value in the read log item; If the target LP record is not found, the log entry being read is determined to be an invalid log entry. If the target LP record is found, then the target LP record and the log item are matched and verified. Log entries that fail the matching verification are identified as invalid log entries; Log entries that pass the matching verification are identified as valid log entries.

[0090] In one specific embodiment of this application, the HP metadata update module is specifically used to query HP records from the HP metadata cache based on the hash in the valid log item; If a corresponding HP record is found, the valid log entries are ignored; If no corresponding HP record is found, add an HP record that matches the valid log entry to the HP metadata cache.

[0091] In one specific embodiment of this application, the HP record of the data block records the hash value, storage pool address, volume logical address, and data deduplication count of the data block. Correspondingly, the metadata comprehensive recording module is specifically used to determine whether the data duplication count in the HP record of a data block is 1; If so, read the first volume logical address and the first storage pool address from the HP record of the data block; Query the address of the first storage pool based on the logical address of the first volume in the LP metadata; If the first storage pool address is not found, after deleting the HP record of the data block in the HP metadata cache, the data block is written to the newly allocated target storage pool address; combining the target volume logical address and the target storage pool address, the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata. If the address of the first storage pool is found, the PL record of the last written data block is inserted into the PL metadata by combining the logical address of the first volume and the address of the first storage pool. The PL record of the current written data block is inserted into the PL metadata by combining the logical address of the target volume. The LP record of the current written data block is inserted into the LP metadata. The data duplication count in the HP record of the data block in the HP metadata cache is incremented by one. If not, then based on the target volume logical address, insert the PL record of this data block in the PL metadata and the LP record of this data block in the LP metadata, and increment the data duplication count in the HP record of the data block in the HP metadata cache.

[0092] In one specific embodiment of this application, the metadata comprehensive recording module is further configured to determine whether there is a corresponding old PL record for the old LP record corresponding to the target volume logical address before inserting LP data into the LP metadata; if so, the old PL record is deleted.

[0093] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below and the data deduplication method described above can be referred to in correspondence.

[0094] See Figure 4 As shown, the electronic device includes: Memory 332 is used to store computer programs; The processor 322 is used to implement the data deduplication method steps of the above method embodiments when executing a computer program.

[0095] For details, please refer to Figure 5 , Figure 5This is a schematic diagram of a specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0096] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0097] The steps in the data deduplication method described above can be implemented by the structure of an electronic device.

[0098] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below and the data deduplication method described above can be referred to in correspondence.

[0099] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data deduplication method described in the above method embodiments.

[0100] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0101] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0102] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0103] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0104] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0105] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data deduplication method, characterized in that, include: Obtain the data block to be written and the target volume logical address, and calculate the target hash value of the data block; In the HP metadata cache, query the HP record of the data block based on the target hash value; If the HP record of the data block is not found, the data block is written to the newly allocated target storage pool address, and the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata cache, in combination with the target volume logical address and the target storage pool address. If the HP record of the data block is found, then the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, and the HP record of the data block in the HP metadata cache is updated. The HP record is a mapping information between the hash value of a data block and the address of the storage pool. The LP record is the mapping information between the volume logical address and the storage pool address of the data block; The PL record contains mapping information between the storage pool address and the volume logical address of the data block.

2. The method according to claim 1, characterized in that, Also includes: If the cache space is insufficient, the target HP record will be evicted from the HP metadata cache; The target log block corresponding to the hash value range to which the hash value in the target HP record falls is determined from the HP metadata log area; wherein, the HP metadata log area includes several log blocks divided according to different hash value ranges; Write the log entry corresponding to the target HP record into the target log block; wherein the log entry includes a hash value, a volume logical address, and a storage pool address.

3. The method according to claim 2, characterized in that, Also includes: Log entries are read from the HP metadata log area in the order of newest to oldest, using a single log block as the reading unit. The read log entries are filtered to obtain the valid log entries; Update the HP metadata cache using the valid log entries.

4. The method according to claim 3, characterized in that, The read log entries are filtered to obtain valid log entries, including: In the LP metadata, LP record queries are performed based on the hash value in the read log item; If the target LP record is not found, the log entry being read is determined to be an invalid log entry. If the target LP record is found, the target LP record and the log item are matched and verified. Log entries that fail the matching verification are identified as invalid log entries; Log entries that pass the matching verification are identified as valid log entries.

5. The method according to claim 3, characterized in that, Updating the HP metadata cache using the valid log entries includes: HP record lookup is performed from the HP metadata cache based on the hash in the valid log entry; If a corresponding HP record is found, the valid log entries are ignored. If no corresponding HP record is found, an HP record matching the valid log entry is added to the HP metadata cache.

6. The method according to any one of claims 1 to 5, characterized in that, The HP record of a data block contains the hash value, storage pool address, volume logical address, and data deduplication count of the data block. Accordingly, if the HP record of the data block is found, the PL record of the data block is inserted into the PL metadata and the LP record of the data block is inserted into the LP metadata, based on the target volume logical address. The HP record of the data block in the HP metadata cache is then updated, including: Determine whether the data duplication count in the HP record of the data block is 1; If so, the first volume logical address and the first storage pool address are read from the HP record of the data block; The address of the first storage pool is queried from the LP metadata based on the logical address of the first volume; If the first storage pool address is not found, after deleting the HP record of the data block in the HP metadata cache, the data block is written to the newly allocated target storage pool address; combining the target volume logical address and the target storage pool address, the HP record of the data block is inserted into the HP metadata cache and the LP record of the data block is inserted into the LP metadata. If the address of the first storage pool is found, then the PL record of the last time the data block was written is inserted into the PL metadata in combination with the logical address of the first volume and the address of the first storage pool. The PL record of the current time the data block is written is inserted into the PL metadata in combination with the logical address of the target volume. The LP record of the current time the data block is written is inserted into the LP metadata. The data duplication count in the HP record of the data block in the HP metadata cache is incremented by one. If not, then, based on the target volume logical address, insert the PL record for this write to the data block into the PL metadata and the LP record for this write to the data block into the LP metadata, and increment the data duplication count in the HP record of the data block in the HP metadata cache by one.

7. The method according to claim 6, characterized in that, Before inserting the LP record into the LP metadata, the following is also included: Determine whether a corresponding old PL record exists for the old LP record corresponding to the logical address of the target volume; If so, delete the old PL record.

8. A data deduplication device, characterized in that, include: The hash calculation module is used to obtain the data block to be written and the target volume logical address, and to calculate the target hash value of the data block; The data deduplication module is used to query the HP record of the data block in the HP metadata cache based on the target hash value; The metadata omission record module is used to write the data block into the newly allocated target storage pool address if no HP record of the data block is found, and to insert the HP record of the data block into the HP metadata cache and the LP record of the data block into the LP metadata cache by combining the target volume logical address and the target storage pool address. The metadata comprehensive recording module is used to, if the HP record of the data block is found, insert the PL record of the data block into the PL metadata and the LP record of the data block into the LP metadata, and update the HP record of the data block in the HP metadata cache, in combination with the target volume logical address. The HP record is a mapping information between the hash value of a data block and the address of the storage pool. The LP record is the mapping information between the volume logical address and the storage pool address of the data block; The PL record contains mapping information between the storage pool address and the volume logical address of the data block.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data deduplication method as described in any one of claims 1 to 7 when executing the computer program.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the data deduplication method as described in any one of claims 1 to 7.

Images