ZNS SSD-based repeated data deletion method and system

Abstract

The invention discloses a repeated data deletion method and system based on ZNS SSD, and belongs to the field of repeated data deletion.The method comprises the steps that a fingerprint index table and a fingerprint updating cache are maintained, three-time opportunity repeated data deletion is achieved, and repeated data search is conducted in the fingerprint index table, the fingerprint updating cache and an address mapping table respectively; an address mapping table, an address reverse mapping table and a reference counting table are maintained, physical space garbage collection / defragmentation and write request merging are achieved, the address mapping table, the address reverse mapping table and the reference counting table conduct garbage collection with partition as granularity when free space is insufficient, and a fingerprint index table is not updated after garbage collection data migration; and instead, the mapping of fingerprints to migration-in physical pages is stored in a fingerprint updating cache, the physical pages are sequentially distributed in partitions by the fingerprint updating cache, normal data write requests and garbage collection data migration write requests are respectively cached, and the write requests which are continuous in sequence are merged and then issued. According to the method, the high-performance and low-cost efficient storage system can be constructed on the basis of the ZNS SSD at relatively low overhead.

Description

Technical Field

[0001] The present invention belongs to the field of deduplication, and more specifically, relates to a deduplication method and system based on ZNS SSDs. Background Art

[0002] With the growth of the global data volume, the cost of storage systems has been increasing continuously. On the one hand, deduplication technology can significantly reduce the storage cost of the system by identifying and avoiding the storage of duplicate data. On the other hand, flash-based solid state drives (SSDs) have been widely used due to their advantages in performance, reliability, etc., and the emerging zoned namespace interface technology (ZNS) can greatly reduce the SSD storage cost by eliminating the need for large-capacity caches and over-provisioned flash space.

[0003] The ZNS interface abstracts the storage device address space into a series of consecutive logical partitions (zones), and its operation characteristics are as follows: the zone needs to be opened first before writing, becoming an open / active zone; the writing of logical pages within the open zone must be in logical address order and cannot be updated in place; when the data needs to be updated, the entire zone must be erased before writing can start again in sequence. It is precisely because of the characteristic of sequential writing within the zone required by ZNS that the use of ZNS SSDs is restricted to a certain extent. And existing deduplication systems are all designed and implemented based on traditional mechanical hard disks and solid state drives that support random writing, and no special design adapted to ZNS SSDs is made for data writing, and the allocation of physical pages is not completely sequential within the zone, that is, existing deduplication systems cannot meet the sequential writing constraints of ZNS SSDs, as Figure 1 shown, which makes the advantages of deduplication technology and the new ZNS SSD storage device unable to be fully utilized simultaneously.

[0004] In addition, traditional deduplication systems do not perform garbage collection. When the free space in the system is insufficient, only the metadata of invalid pages is modified to make them allocable, resulting in fragmentation problems in its IO write requests and a decline in system performance, as Figure 2 shown; moreover, garbage collection is no longer performed inside the ZNS SSD, and it entrusts garbage collection to the host side. The deduplication system must take the responsibility of garbage collection, which is also a problem faced by traditional deduplication systems.

[0005] Furthermore, garbage collection requires data migration. If the fingerprint index table is not updated after data migration, it will lead to the fingerprint pointing to the wrong physical page, that is, the fingerprint index is inconsistent, as Figure 3As shown, this will lead to a decrease in the deduplication rate; if the fingerprint index table is updated, since ZNS SSD requires erasure at the partition granularity, garbage collection must also be performed at the partition granularity. However, the fingerprints corresponding to the physical pages within a partition are discretely distributed, and the update operation of the fingerprint index table will introduce significant fingerprint read / write overhead, thereby reducing system performance.

[0006] Generally speaking, the existing deduplication technologies are still unable to effectively build a high-performance and low-cost efficient storage system based on ZNS SSD with low overhead. Summary of the Invention

[0007] In view of the deficiencies and improvement requirements of the prior art, the present invention provides a deduplication method and system based on ZNS SSD, aiming to effectively build a high-performance and low-cost efficient storage system based on ZNS SSD with low overhead.

[0008] To achieve the above object, according to one aspect of the present invention, there is provided a deduplication method based on ZNS SSD, including:

[0009] Initialization: Abstract the physical space of ZNS SSD into multiple partitions, allocate one partition as the data write partition, and initialize the data write pointer WP to point to the first physical page of this partition;

[0010] Write request processing: Split the to-be-processed IO write request into data block IO requests according to the physical page size of ZNS SSD, query whether the data blocks corresponding to each data block IO request are duplicate data blocks. If so, update the metadata of the data block to complete the processing of the corresponding data block IO request. Otherwise, execute the write request merging step to issue or cache the corresponding data block IO request; the write request merging step includes:

[0011] S1: Select the physical page PPNnow pointed to by the current data write pointer WP as the physical page for writing the current data block. If PPNnow is the last physical page of the data write partition, allocate an idle partition as the new data write partition and make WP point to the first physical page of this partition; otherwise, make WP point to the next physical page of PPNnow;

[0012] S2: Update the metadata of PPNnow and the logical page LPNnow corresponding to the current data IO request;

[0013] S3: Issue or cache the current data block IO request;

[0014] Among them, the metadata includes: an address mapping table for recording the mapping relationship between logical pages and physical pages, a reference count table for recording the reference count of physical pages, and a fingerprint index table for recording the fingerprints of data blocks and the corresponding physical pages and logical pages.

[0015] Furthermore, the initialization further includes: creating an IO request cache IO_Buffer of a specified size in memory;

[0016] And, step S3 includes:

[0017] S31: If IO_Buffer is empty, or the physical page PPNlast of the previous data block IO request cached in IO_Buffer satisfies PPNnow = PPNlast + 1, then go to S32; otherwise, go to S34;

[0018] S32: Cache the current data block IO request into IO_Buffer;

[0019] S33: If IO_Buffer is full after caching the current data block IO request into IO_Buffer, then merge the data block IO requests in IO_Buffer and issue them, and step S3 ends;

[0020] S34: Merge the data block IO requests in IO_Buffer and issue them, and then cache the current data block IO request into IO_Buffer, and step S3 ends.

[0021] Furthermore, the N physical pages at the end of the data writing partition are fingerprint pages for recording the fingerprints of the data blocks stored in the data writing partition; N is a preset positive integer;

[0022] And, the deduplication method based on ZNS SSD further includes: each time a new data writing partition is allocated, creating temporary fingerprint pages in memory that correspond one-to-one with the N fingerprint pages for this data writing partition;

[0023] And, step S2 includes:

[0024] S21: Query in the address mapping table whether the physical page corresponding to LPNnow exists. If so, decrement the reference count corresponding to this physical page in the reference count table, update the physical page corresponding to LPNnow in the address mapping table to PPNnow, and go to S22; otherwise, insert an entry "LPNnow, PPNnow" into the address mapping table and go to S22;

[0025] S22: Increment the reference count of PPNnow in the reference count table and write the fingerprint FP of the current data block into the temporary fingerprint page;

[0026] S23: If the fingerprint FP does not exist in the fingerprint index table, insert the entry "FP, PPNnow, LPNnow" into the fingerprint index table; otherwise, update the physical page and logical page corresponding to FP in the fingerprint index table to PPNnow and LPNnow respectively;

[0027] Moreover, the write request merging step further includes after S3:

[0028] If PPNnow is not the last physical page of the current data writing partition, the write request merging step ends;

[0029] If PPNnow is the last physical page of the current data writing partition, create a first IO request to sequentially write the temporary fingerprint page to the corresponding fingerprint page; if the current data block IO request has not been issued yet, merge the first IO request with all the IO requests in the IO_Buffer and then issue it, and the write request merging step ends; if the current data block IO request has already been issued, sequentially issue the first IO request, and the write request merging step ends.

[0030] Furthermore, the metadata further includes an address reverse mapping table for recording the mapping relationship from physical pages to logical pages;

[0031] Moreover, step S21 further includes: inserting the entry "PPNnow, LPNnow" into the address reverse mapping table;

[0032] Moreover, the initialization further includes: allocating a fingerprint update cache of a preset size in the memory for recording the mapping relationship from data block fingerprints to physical pages after data migration in garbage collection; allocating a partition as the GC writing partition, and initializing the GC write pointer GC_WP to point to the first physical page of the current GC writing partition;

[0033] Moreover, the deduplication method based on ZNS SSD further includes: when the number of free partitions is less than a preset threshold, perform the garbage collection step; the garbage collection step includes:

[0034] T1: Traverse all partitions, select the partition with the most physical pages with a reference count of 0 as the partition to be migrated, and read the fingerprint page at the end of the partition to be migrated;

[0035] T2: Traverse the physical pages in the partition to be migrated, query the reference count REFgc of the current physical page PPNgc in the reference count table, if REFgc = 0, continue to traverse, otherwise read the data of the current physical page, and after sequentially performing the free page allocation step, the fingerprint update cache update step, and the metadata update step, create a second IO request to write the current physical page, and issue or cache the second IO request;

[0036] T3: Erase all the pages in the partition to be migrated, clear the reference counts of all physical pages of the partition to be migrated in the reference count table, and then mark the partition to be migrated as a free partition;

[0037] The free page allocation steps include:

[0038] Select the physical page PPNin pointed to by the GC write pointer GC_WP as the physical page to which the currently traversed physical page is migrated. If PPNin is the last physical page in the current GC write partition, allocate a free partition as the new GC write partition and make GC_WP point to the first physical page of this partition; otherwise, make GC_WP point to the next physical page of PPNin;

[0039] The fingerprint update cache update steps include:

[0040] Obtain the fingerprint FPGc of PPNgc from the fingerprint page at the end of the partition to be migrated, and insert the entry "FPGc, PPNin" into the fingerprint update cache; if the fingerprint update cache is full after inserting this entry, eliminate the oldest entry in the fingerprint update cache according to the cache time;

[0041] The metadata update steps include:

[0042] Query the logical address LPNgc corresponding to PPNgc in the address reverse mapping table, update the physical page corresponding to LPNgc in the address mapping table to PPNin, delete the entry "PPNgc, LPNgc" in the address reverse mapping table, insert the entry "PPNin, LPNgc", and add REFgc to the reference count corresponding to PPNin in the reference count table.

[0043] Furthermore, the N physical pages at the end of the GC write partition are fingerprint pages for recording the fingerprints of data blocks stored in the GC write partition;

[0044] And the initialization further includes: creating a GC_IO_Buffer of a specified size in memory;

[0045] And the deduplication method based on ZNS SSD further includes: each time a new GC write partition is allocated, creating temporary fingerprint pages corresponding one-to-one with the N fingerprint pages for this GC write partition in memory;

[0046] And in step T2, the second IO request is issued or cached through the garbage collection write request merging step; the garbage collection write request merging step includes:

[0047] Append and write FPGc to the temporary fingerprint page of the current GC write partition;

[0048] Cache the IO requests in the GC_IO_Buffer; if the GC_IO_Buffer is full, merge the requests in the GC_IO_Buffer and then issue them.

[0049] If PPNin is not the last data page in the current GC write partition, the garbage collection write request merging step ends; otherwise, create a third IO request to sequentially write the corresponding temporary fingerprint page to the current GC write partition. If the second IO request has not been issued yet, merge the third IO request with all the requests in the GC_IO_Buffer and then issue them; if the third IO request has already been issued, issue the third IO request sequentially.

[0050] Furthermore, during the write request processing, check whether each data block is a duplicate data block through triple - pass deduplication. The triple - pass deduplication includes:

[0051] D1: Search for the physical page PPN and logical page LPN corresponding to the fingerprint FP of the current data in the fingerprint index table. If the search is successful, go to D2; otherwise, determine that the current data block is not a duplicate data block, and the triple - pass deduplication ends.

[0052] D2: After reading the data corresponding to PPN, compare it byte - by - byte with the current data block. If they are exactly the same, determine that the current data block is a duplicate data block and go to D3; otherwise, go to D5.

[0053] D3: Query the logical page LPNnow corresponding to the IO request of the current data block in the address mapping table to obtain the corresponding physical page PPNnow. If the query is successful, decrement the reference count corresponding to PPNnow in the reference count table and go to D4; otherwise, directly go to D4.

[0054] D4: Update the physical page corresponding to LPNnow in the address mapping table to PPN, insert the "PPN, LPNnow" entry in the address reverse mapping table, and increment the reference count corresponding to PPN in the reference count table. The triple - pass deduplication ends.

[0055] D5: Traverse and query in the fingerprint update cache in the order from new to old according to the cache time to check whether FP exists. If it exists, go to D6; otherwise, go to D7.

[0056] D6: Obtain the physical page newPPN corresponding to FP in the fingerprint update cache, read the data in newPPN, and compare it byte - by - byte with the current data block. If they are exactly the same, determine that the current data block is a duplicate data block and go to D7; otherwise, go to D9.

[0057] D7: Query the physical page PPNnow corresponding to LPNnow in the address mapping table. If the query is successful, decrement the reference count corresponding to PPNnow in the reference count table and proceed to D8; otherwise, directly proceed to D8;

[0058] D8: Update the physical page corresponding to LPNnow in the address mapping table to newPPN, insert the entry "newPPN, LPNnow" into the address reverse mapping table, increment the reference count corresponding to newPPN in the reference count table, and update the physical page corresponding to FP in the fingerprint index table to newPPN. The triple - chance deduplication ends.

[0059] D9: Query the physical page latentPPN corresponding to LPN in the address mapping table. If the search is successful, proceed to D10; otherwise, determine that the current data block is not a duplicate data block and the triple - chance deduplication ends;

[0060] D10: Read the data corresponding to latentPPN and compare it byte - by - byte with the current data block. If they are exactly the same, determine that the current data block is a duplicate data block and proceed to D11; otherwise, determine that the current data block is not a duplicate data block and the triple - chance deduplication ends;

[0061] D11: Query the physical page corresponding to LPNnow in the address mapping table. If the query is successful, decrement the reference count corresponding to this physical page in the reference count table and proceed to D12; otherwise, directly proceed to D12;

[0062] D12: Update the physical page corresponding to LPNnow in the address mapping table to latentPPN, insert the entry "latentPPN, LPNnow" into the address reverse mapping table, increment the reference count corresponding to latentPPN in the reference count table, and update the physical page corresponding to FP in the fingerprint index table to latentPPN. The triple - chance deduplication ends.

[0063] According to another aspect of the present invention, there is provided a computer program product, including a computer program; when the computer program is executed by a processor, the above - described deduplication method based on ZNS SSD provided by the present invention is implemented.

[0064] According to another aspect of the present invention, there is provided a computer - readable storage medium, including a stored computer program, when the computer program is executed by a processor, controlling the device where the computer - readable storage medium is located to execute the above - described deduplication method based on ZNS SSD provided by the present invention.

[0065] According to another aspect of the present invention, there is provided a deduplication controller based on ZNS SSD, including:

[0066] A computer-readable storage medium for storing a computer program;

[0067] And a processor for reading the computer program stored in the computer-readable storage medium and executing the above-mentioned deduplication method based on ZNS SSD provided by the present invention.

[0068] According to another aspect of the present invention, there is provided a storage system, including: a ZNS SSD and the above-mentioned deduplication controller based on ZNS SSD provided by the present invention.

[0069] Generally speaking, through the above technical solutions conceived by the present invention, the following beneficial effects can be achieved:

[0070] (1) The present invention abstracts the physical space of the ZNS SSD into partitions. During deduplication, a physical page sequential write allocation strategy is implemented through data writing to partitions and data write pointers, meeting the in-partition sequential write constraint of the ZNS SSD, enabling the deduplication technology to be adapted to the ZNS SSD, thereby being able to fully utilize the advantages of the deduplication technology and the new storage device of the ZNS SSD, expanding the storage capacity and reducing the system cost. Generally speaking, the present invention can effectively construct a high-performance and low-cost efficient storage system based on the ZNS SSD with low overhead.

[0071] (2) On the basis of implementing the physical page sequential write allocation strategy, the present invention uses the IO_Buffer to cache normal data write requests, thereby being able to realize write request IO merging by taking advantage of the sequential IO request opportunities created by the sequential write allocation strategy, merging sequential and continuous write requests and then issuing them, alleviating the write request fragmentation problem and improving the system IO performance.

[0072] (3) The present invention performs garbage collection at the partition granularity to regularly recycle invalid data and its corresponding invalid metadata, making the deduplication compatible with the ZNS SSD; on this basis, the present invention maintains a fingerprint update cache. After garbage collection data migration, instead of updating the fingerprint index table, it saves the mapping of the fingerprint to the migrated physical page in the fingerprint update cache, thereby being able to eliminate the fingerprint index update overhead, improve the system's IO performance, and use the fingerprint index table, fingerprint update cache, and address mapping table to implement three-chance deduplication to ensure a high deduplication rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 A schematic diagram of randomly allocating physical pages for an existing deduplication system;

[0074] Figure 2 Schematic diagram of the fragmentation problem of the IO write requests in the existing deduplication system

[0075] Figure 3 Schematic diagram of the inconsistency of the fingerprint index caused by not updating the fingerprint index after garbage collection in the existing deduplication system

[0076] Figure 4 Schematic diagram of the system architecture of the high-performance and low-cost deduplication method based on ZNS SSD provided by the embodiments of the present invention Detailed implementation manners

[0077] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other

[0078] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects and do not necessarily need to describe a specific order or sequence

[0079] In order to adapt the deduplication technology to ZNS SSD, alleviate the fragmentation problem of IO write requests, and avoid the loss or performance degradation of deduplication caused by garbage collection, and build a high-performance and low-cost efficient storage system with low overhead, the present invention provides a deduplication method and system based on ZNS SSD. The overall concept is as follows: the deduplication system abstractly divides the physical space into partitions, and sequentially allocates physical pages within the partitions to meet the constraint of sequential writing within the ZNS SSD partitions; caches normal data write requests and GC migration write requests respectively, so as to merge and issue sequential requests by means of the sequential write allocation strategy and the sequential IO request opportunities created by garbage collection, and alleviate the fragmentation problem of write requests; when the number of idle partitions is less than the threshold, garbage collection is performed at the partition level. After the garbage collection data is migrated, the fingerprint index table is not updated, but the fingerprint of the migrated physical page is mapped to the physical page it migrates into, and is stored in the fingerprint update cache with the LRU strategy; three-chance deduplication is adopted to sequentially query the same data in the fingerprint index table and the address mapping table of the fingerprint update cache respectively, so as to ensure accurate judgment of duplicate data blocks even when the fingerprint indexes are inconsistent

[0080] Based on the above technical concept, as Figure 1As shown, in an alternative embodiment of the present invention, namely Embodiment 1, a fingerprint index table, a fingerprint update cache, an address mapping table, an address reverse mapping table, and a reference count table are maintained; the fingerprint index table is used to record the mapping relationship between fingerprints and physical pages and logical pages. During the first deduplication query process, if the fingerprint is not found, the mapping entry of the fingerprint to the physical page and the logical page is inserted. If the fingerprint is found but the data verification fails, the entry is updated subsequently; the fingerprint update cache is used to record the mapping relationship between fingerprints and new physical pages after garbage collection data migration, and the fingerprint update cache adopts the LRU strategy for elimination, and the eliminated mapping entries are directly discarded; the address mapping table is used to record the mapping relationship between logical pages and physical pages, the address reverse mapping table is used to record the mapping relationship between physical pages and logical pages, and the reference count table is used to record the reference count of each physical page.

[0081] Embodiment 1 of the present invention also implements triple-chance deduplication, and duplicates are queried separately in the fingerprint index table, the fingerprint update cache, and the address mapping table: if any one of the three queries is successful and the data verification is successful, it indicates the existence of duplicates; if the first query fails, it indicates that there must be no duplicates. If the query is successful but the data verification fails, it indicates that the physical page corresponding to the fingerprint may have been migrated by garbage collection without updating the corresponding fingerprint index, and thus a second query is performed in the fingerprint update cache; if the second query fails, it indicates that the entry corresponding to the fingerprint may have been eliminated from the cache by the LRU strategy, and a third query is further performed in the address mapping table using the logical address corresponding to the fingerprint in the fingerprint index table. In addition, the second query only occurs when the hash algorithm itself causes a conflict, and the probability is negligible; if the third query fails, it indicates that there must be no duplicates.

[0082] Embodiment 1 of the present invention also implements physical space garbage collection / de-fragmentation and write request merging functions.

[0083] For physical space garbage collection / de-fragmentation, when the number of free partitions is less than the threshold, the valid pages in the partition with the most invalid pages are migrated to the GC write partition, the metadata is modified accordingly, and the migrated partition is erased and reset; in the present invention, the fingerprint index table is not updated after data migration to eliminate the fingerprint index update overhead. However, to ensure a high deduplication rate, the mapping relationship between fingerprints and new physical pages after data migration is saved in the fingerprint update cache for the second deduplication query; and to save the entry information in the fingerprint update cache, the fingerprint corresponding to the physical page needs to be obtained. In the present invention, several fixed fingerprint pages are reserved at the end of each partition to store the fingerprint values of the physical pages in this partition. During garbage collection, these fingerprint pages are read first, and the corresponding fingerprint values are obtained according to the position of the physical page in the partition.

[0084] The write request merging function includes steps such as free page allocation, metadata update, IO request caching, and temporary fingerprint page caching. Free page allocation is the key for the deduplication system to adapt to ZNS SSD. Normal data writing and GC migration writing are carried out within the data writing partition and the GC writing partition respectively. The data write pointer and the GC write pointer are used to indicate the next physical page to be allocated for normal data writing and GC migration writing respectively. After the physical page is allocated, the write pointer moves sequentially within the partition. After the partition is full, a free partition is selected again for writing. IO request caching is the key to alleviating the fragmentation problem of IO write requests. Due to the sequential writing feature, there are a large number of sequential continuous IO write requests. The normal write requests and GC migration write requests are temporarily stored in the IO Buffer and the GC_IO Buffer respectively, and the sequential continuous IO write requests are merged and then issued.

[0085] The operation of abstracting the physical space into partitions, the initialization of the fingerprint index table, the fingerprint update cache, the address mapping table, the address reverse mapping table, and the reference count table, as well as the initialization of the data writing partition, the data write pointer WP, the GC writing partition, and the GC write pointer GC_WP, are all completed at the initial moment.

[0086] When an IO write request arrives, data chunking is performed first. Specifically, according to the physical page size in the ZNS SSD (such as 4KB), the original IO write request is divided into one or more data chunk IO requests. Secondly, a preset hash function (such as MD5, SHA-1, etc.) is used for fingerprint calculation. Thirdly, the three-chance deduplication is used to query whether each data chunk is a duplicate data chunk. Finally, for unique data chunks (i.e., non-duplicate data chunks), write request merging is performed.

[0087] Specifically, the steps of the three-chance deduplication are as follows:

[0088] (1) Query whether the fingerprint value FP of the current data chunk exists in the fingerprint index table. If it exists, enter the first data verification step (2). Otherwise, it means that the current data chunk is a unique data chunk, and enter the write request merging step.

[0089] (2) The physical page PPN and the logical page LPN corresponding to FP in the fingerprint index table are obtained, and a read request is issued to read the data corresponding to PPN, and it is compared byte by byte with the current data chunk. If they are exactly the same, it means that the current data chunk is a duplicate data chunk, and transfer to step (3) to update the metadata. Otherwise, transfer to step (4) to perform the second deduplication query.

[0090] (3) Query whether the physical page corresponding to the logical page LPNnow of the current data block IO request exists in the address mapping table. If it exists, it indicates that the current data block IO request is an update write to LPNnow. Then, decrement the reference count corresponding to this physical page in the reference count table, update the physical page corresponding to LPNnow to PPN in the address mapping table, insert the entry "PPN, LPNnow" into the address reverse mapping table, and increment the reference count corresponding to PPN in the reference count table. The process ends. If the physical page corresponding to LPNnow does not exist in the address mapping table, it indicates that no data block has been written to LPNnow before. Then, insert the entry "LPNnow, PPN" into the address mapping table, insert the entry "PPN, LPNnow" into the address reverse mapping table, and increment the reference count corresponding to PPN in the reference count table. The process ends;

[0091] (4) Traverse and query in the fingerprint update cache (organized by data structures such as linked lists and priority queues) in order from new to old according to the cache time to check if the fingerprint value FP exists. If it exists, stop traversing and transfer to step (5) for the second data verification. Otherwise, transfer to step (7) for the third duplicate data query;

[0092] (5) Send a read request to the physical page newPPN corresponding to FP in the fingerprint update cache to read the data corresponding to newPPN, and compare it byte by byte with the current data block. If they are exactly the same, it indicates that the current data block is a duplicate data block, and then transfer to step (6) for metadata update. Otherwise, it indicates that the current data block is a unique data block, and transfer to step (7) for the third duplicate data query;

[0093] (6) Query whether the physical page corresponding to LPNnow exists in the address mapping table. If it exists, decrement the reference count corresponding to this physical page in the reference count table, update the physical page corresponding to LPNnow to newPPN in the address mapping table, insert the entry "newPPN, LPNnow" into the address reverse mapping table, increment the reference count corresponding to newPPN in the reference count table, and update the physical page corresponding to FP to newPPN in the fingerprint index table. The process ends. Otherwise, insert the entry "LPNnow, PPN" into the address mapping table, insert the entry "PPN, LPNnow" into the address reverse mapping table, increment the reference count corresponding to PPN in the reference count table, and update the physical page corresponding to FP to newPPN in the fingerprint index table. The process ends;

[0094] (7) Query whether the entry corresponding to LPN exists in the address mapping table. If it exists, transfer to step (8) for the third data verification. Otherwise, it indicates that the current data block is a unique data block, and enter the write request merging step;

[0095] (8) Send a read request to the physical page latentPPN corresponding to the LPN in the address mapping table to read the data corresponding to latentPPN, and compare it byte by byte with the current data block. If they are exactly the same, it means the current data block is a duplicate data block, and then go to step (9) for metadata update; otherwise, it means the current data block is a unique data block, and enter the write request merging step;

[0096] (9) Query whether the physical page corresponding to LPNnow exists in the address mapping table. If it exists, decrement the reference count corresponding to this physical page in the reference count table, update the physical page corresponding to LPNnow in the address mapping table to latentPPN, insert the entry "latentPPN, LPNnow" into the address reverse mapping table, increment the reference count corresponding to latentPPN in the reference count table, and update the physical page corresponding to FP in the fingerprint index table to latentPPN, and the process ends; otherwise, insert the entry "LPNnow, latentPPN" into the address mapping table, insert the entry "latentPPN, LPNnow" into the address reverse mapping table, increment the reference count corresponding to latentPPN in the reference count table, and update the physical page corresponding to FP in the fingerprint index table to latentPPN, and the process ends.

[0097] Specifically, the steps of write request merging are as follows:

[0098] (1) Select the physical page PPNnow pointed to by the data write pointer WP. If PPNnow is the last data page in the partition, traverse the partition and select an idle partition to be marked as the data write partition, and point WP to the first physical page of this partition; otherwise, increment WP by 1 to point to the next physical page of PPNnow; enter step (2) to execute the metadata update step;

[0099] (2) Query whether the physical page corresponding to LPNnow exists in the address mapping table. If it exists, decrement the reference count corresponding to this physical page in the reference count table, update the physical page corresponding to LPNnow in the address mapping table to PPNnow, insert the entry "PPNnow, LPNnow" into the address reverse mapping table, increment the reference count corresponding to PPNnow in the reference count table, append FP to the temporary fingerprint page. If FP does not exist in the fingerprint index table, insert "FP, PPNnow, LPNnow" into the fingerprint index table; otherwise, update the physical page and logical page corresponding to FP to PPNnow and LPNnow respectively, and go to step (3) to execute the IO cache step;

[0100] (3) Compare the physical page PPNnow of the current data block IO request with the cached previous physical page PPNlast of the data block IO request. If the data IO_Buffer is empty or PPNnow = PPNlast + 1, cache the current data block IO request into the IO_Buffer, and update PPNlast to PPNnow. If the data IO_Buffer is full after caching the current IO request, merge the requests in the IO_Buffer and then issue them; if the IO_Buffer is not empty and PPNnow is not equal to PPNlast + 1, merge the requests in the data IO_Buffer and then issue them, then cache the current IO request into the data IO_Buffer, and update PPNlast to PPNnow; go to step (4) to execute the temporary fingerprint page processing step;

[0101] (4) If PPNnow is not the last data page of the partition it belongs to, the process ends; otherwise, create a first IO request to sequentially write the corresponding temporary fingerprint page to the reserved fingerprint page of the partition. If the current data block IO request has not been issued yet, merge the created first IO request with all the requests in the IO_Buffer and then issue them; otherwise, issue the created first IO request sequentially, and the process ends.

[0102] When the number of free partitions is less than the threshold, garbage collection will be triggered to recycle invalid pages, that is, pages with a reference count of 0, to clear free space. During garbage collection, for the valid physical pages in the selected partition, they need to be migrated to the GC write partition, and the fingerprint index table is not updated after migration to save the update overhead of the fingerprint index table. Since this embodiment proposes three - chance deduplication, even when the fingerprint indexes are inconsistent, it can accurately determine whether a data block is a duplicate data block. Specifically, the steps of garbage collection are as follows:

[0103] (1) Traverse all partitions, select the partition with the largest number of invalid pages (physical pages with a reference count of 0) as the partition to be migrated, issue a read request to read the fingerprint page at the end of the partition to be migrated, and go to step (2) to read the page to be migrated;

[0104] (2) Traverse the physical pages of the partition to be migrated, query the reference count REFgc of the current physical page PPNgc in the reference count table. If REFgc is equal to 0, continue traversing; otherwise, issue a read request to read the data of this page, and go to step (3) to execute the free page allocation step;

[0105] (3) Select the physical page PPNin pointed to by the GC write pointer WP_GC. If PPNin is the last data page in the partition, traverse the partition and select an idle partition and mark it as the GC write partition, and point WP_GC to the first physical page of this partition; otherwise, increment WP_GC by 1 to point to the next physical page; go to step (4) to perform the fingerprint update cache update step;

[0106] (4) Obtain the corresponding fingerprint value FPgc from the fingerprint page according to the position of the physical page PPNin in the partition, insert the entry "FPgc, PPNin" into the fingerprint update cache. If the fingerprint update cache is full, eliminate the oldest entry according to the cache time, and go to step (5) for metadata update;

[0107] (5) Query the logical address LPNgc corresponding to PPNgc (there may be multiple LPNgcs) in the address reverse mapping table, update all physical pages corresponding to LPNgc in the address mapping table to PPNin, delete "PPNgc, LPNgc" in the address reverse mapping table, and insert the entry "PPNin, LPNgc", and increment the reference count corresponding to PPNin in the reference count table by REFgc, and additionally write FPgc into the temporary fingerprint page migrated into the partition, and go to step (6) to perform the garbage collection write request merging step;

[0108] (6) Create a second IO request to write the data to be read into the physical page PPNin, cache this second IO request into GC_IO_Buffer. If GC_IO_Buffer is full, merge the requests in GC_IO_Buffer and then issue them, and go to step (7) to perform the garbage collection temporary fingerprint page processing step;

[0109] (7) If PPNin is not the last data page in the partition, return to step (2) to continue traversing until the end and then go to step (8); otherwise, create a third IO request to sequentially write the temporary fingerprint page to the reserved fingerprint page of the partition. If the current second IO request has not been issued yet, merge the created third IO request with all requests in GC_IO_Buffer and then issue them, otherwise issue the created third IO request sequentially, return to step (2) to continue traversing until the end and then go to step (8);

[0110] (8) Issue an erase command to erase the entire migrated partition, and clear the reference counts of all physical pages in the partition in the reference count table, and mark the partition as an idle partition, and the process ends.

[0111] Generally speaking, the deduplication method based on ZNS SSD provided in this embodiment adapts the deduplication technology to ZNS SSD, implements a physical page sequential write allocation strategy in the deduplication system to meet the in-partition sequential write constraint of ZNS SSD, enables the deduplication system to be compatible with ZNS SSD, and can give full play to the advantages of both, expanding the storage capacity and reducing the system cost; by virtue of the sequential write allocation strategy and the sequential IO request opportunities created by garbage collection, write request IO merging is realized, normal data write requests and GC migration write requests are cached respectively, and the sequential and continuous write requests are merged and then issued, alleviating the problem of write request fragmentation and improving the IO performance of the system. In addition, this embodiment implements a garbage collection strategy for the deduplication layer at the partition granularity to regularly recycle invalid data and its corresponding invalid metadata. After the garbage collection data migration, the fingerprint index table is not updated, but the mapping from the fingerprint to the migrated physical page is saved in the fingerprint update cache to eliminate the fingerprint index update overhead, improve the IO performance of the system, and use the fingerprint index table, fingerprint update cache and address mapping table to implement three-opportunity deduplication to ensure a high deduplication rate.

[0112] Embodiment 2:

[0113] A computer program product includes a computer program; when the computer program is executed by a processor, it implements the deduplication method based on ZNS SSD provided in the above Embodiment 1.

[0114] Embodiment 3:

[0115] A computer-readable storage medium includes a stored computer program; when the computer program is executed by a processor, it controls the device where the computer-readable storage medium is located to execute the deduplication method based on ZNS SSD provided in the above Embodiment 1.

[0116] Embodiment 4:

[0117] A deduplication controller based on ZNS SSD includes:

[0118] A computer-readable storage medium for storing a computer program;

[0119] And a processor for reading the computer program stored in the computer-readable storage medium and executing the deduplication method based on ZNS SSD provided in the above Embodiment 1.

[0120] Embodiment 5:

[0121] A storage system includes: a ZNS SSD and the deduplication controller based on ZNS SSD provided in the above Embodiment 4.

[0122] Those skilled in the art can easily understand that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A deduplication method based on ZNS SSD, characterized in that, Including: Initialization: Abstract the physical space of the ZNS SSD into multiple partitions, allocate one partition as the data write partition, and initialize the data write pointer WP to point to the first physical page of this partition; Write request processing: Split the to-be-processed IO write request into data block IO requests according to the physical page size of the ZNS SSD, query whether the data blocks corresponding to each data block IO request are duplicate data blocks. If so, update the metadata of the data block to complete the processing of the corresponding data block IO request. Otherwise, execute the write request merging step to issue or cache the corresponding data block IO request; The write request merging step includes: S1: Select the physical page PPNnow pointed to by the current data write pointer WP as the physical page for writing the current data block. If PPNnow is the last physical page of the data write partition, allocate an idle partition as the new data write partition and make WP point to the first physical page of this partition; otherwise, make WP point to the next physical page of PPNnow; S2: Update the metadata corresponding to PPNnow and the logical page LPNnow corresponding to the current data IO request; S3: Issue or cache the current data block IO request; Among them, the metadata includes: an address mapping table for recording the mapping relationship between logical pages and physical pages, a reference count table for recording the reference count of physical pages, and a fingerprint index table for recording the fingerprints of data blocks and the corresponding physical pages and logical pages.

2. The deduplication method based on ZNS SSD according to claim 1, characterized in that The initialization further includes: creating an IO request cache IO_Buffer with a specified size in memory; And, step S3 includes: S31: If IO_Buffer is empty, or the physical page PPNlast of the previous data block IO request cached in IO_Buffer satisfies PPNnow = PPNlast + 1, then go to S32; otherwise, go to S34; S32: Cache the current data block IO request to IO_Buffer; S33: If IO_Buffer is full after caching the current data block IO request to IO_Buffer, then merge and issue the data block IO requests in IO_Buffer, and step S3 ends; S34: Merge and issue the data block IO requests in IO_Buffer, and then cache the current data block IO request to IO_Buffer, and step S3 ends.

3. The deduplication method based on ZNS SSD according to claim 2, wherein The N physical pages at the end of the data write partition are fingerprint pages for recording the fingerprints of the data blocks stored in the data write partition; N is a preset positive integer; And, the deduplication method based on ZNS SSD further includes: each time a new data write partition is allocated, create temporary fingerprint pages in memory for this data write partition that correspond one-to-one with the N fingerprint pages; And, the step S2 includes: S21: Query whether the physical page corresponding to LPNnow exists in the address mapping table. If it does, decrement the reference count corresponding to this physical page in the reference count table by 1, update the physical page corresponding to LPNnow in the address mapping table to PPNnow, and proceed to S22; otherwise, insert an entry "LPNnow, PPNnow" into the address mapping table and proceed to S22; S22: Increment the reference count of PPNnow in the reference count table by 1, and write the fingerprint FP of the current data block into the temporary fingerprint page; S23: If the fingerprint FP does not exist in the fingerprint index table, insert an entry "FP, PPNnow, LPNnow" into the fingerprint index table; otherwise, update the physical page and logical page corresponding to FP in the fingerprint index table to PPNnow and LPNnow respectively; Moreover, after S3, the write request merging step further includes: If PPNnow is not the last physical page of the current data writing partition, the write request merging step ends; If PPNnow is the last physical page of the current data writing partition, create a first IO request to sequentially write the temporary fingerprint page to the corresponding fingerprint page; if the current data block IO request has not been issued yet, merge the first IO request with all the IO requests in IO_Buffer and then issue it, and the write request merging step ends; if the current data block IO request has already been issued, sequentially issue the first IO request, and the write request merging step ends.

4. The deduplication method based on ZNS SSD according to claim 3, wherein The metadata further includes an address reverse mapping table for recording the mapping relationship from physical pages to logical pages; Moreover, step S21 further includes: inserting an entry "PPNnow, LPNnow" into the address reverse mapping table; Moreover, the initialization further includes: allocating a fingerprint update cache of a preset size in the memory for recording the mapping relationship from data block fingerprints to physical pages after data migration in garbage collection; allocating a partition as the GC writing partition, and initializing the GC write pointer GC_WP to point to the first physical page of the current GC writing partition; Moreover, the deduplication method based on ZNS SSD further includes: when the number of free partitions is less than a preset threshold, performing a garbage collection step; the garbage collection step includes: T1: Traverse all partitions, select the partition with the most physical pages with a reference count of 0 as the partition to be migrated, and read the fingerprint page at the end of the partition to be migrated; T2: Traverse the physical pages in the partition to be migrated, query the reference count REFgc of the current physical page PPNgc in the reference count table. If REFgc = 0, continue traversing; otherwise, read the data of the current physical page, and after sequentially performing the free page allocation step, the fingerprint update cache update step, and the metadata update step, create a second IO request to write the current physical page, and issue or cache the second IO request; T3: Erase the entire partition to be migrated, clear the reference counts of all physical pages in the partition to be migrated in the reference count table, and then mark the partition to be migrated as a free partition; The free page allocation step includes: Select the physical page PPNin pointed to by the GC write pointer GC_WP as the physical page to which the currently traversed physical page is migrated. If PPNin is the last physical page within the current GC write partition, allocate an idle partition as the new GC write partition and make GC_WP point to the first physical page of this partition; otherwise, make GC_WP point to the next physical page of PPNin; The fingerprint update cache update step includes: Obtain the fingerprint FPGc of PPNgc from the fingerprint page at the end of the migrated partition, and insert the entry "FPGc, PPNin" into the fingerprint update cache; if the fingerprint update cache is full after inserting this entry, eliminate the oldest entry in the fingerprint update cache according to the cache time; The metadata update step includes: Query the logical address LPNgc corresponding to PPNgc in the address reverse mapping table, update the physical page corresponding to LPNgc in the address mapping table to PPNin, delete the entry "PPNgc, LPNgc" in the address reverse mapping table, and insert the entry "PPNin, LPNgc", and increment the reference count corresponding to PPNin in the reference count table by REFgc.

5. The deduplication method based on ZNS SSD according to claim 4, wherein The N physical pages at the end of the GC write partition are fingerprint pages for recording the fingerprints of the data blocks stored within the GC write partition; Moreover, the initialization further includes: creating an IO request cache GC_IO_Buffer of a specified size in memory; Moreover, the deduplication method based on ZNS SSD further includes: each time a new GC write partition is allocated, creating temporary fingerprint pages in memory that correspond one-to-one with the N fingerprint pages for this GC write partition; Moreover, in step T2, the second IO request is issued or cached through the garbage collection write request merging step; the garbage collection write request merging step includes: Append and write FPGc to the temporary fingerprint page of the current GC write partition; Cache the IO request into GC_IO_Buffer; if GC_IO_Buffer is full, then merge the requests in GC_IO_Buffer and issue them; If PPNin is not the last data page of the GC write partition it belongs to, then the garbage collection write request merging step ends; otherwise, create a third IO request to sequentially write the corresponding temporary fingerprint page to the current GC write partition. If the second IO request has not been issued yet, then merge the third IO request with all the requests in GC_IO_Buffer and issue them; if the third IO request has already been issued, then issue the third IO request sequentially.

6. The deduplication step based on ZNS SSD as claimed in claim 5, wherein In the write request processing, query whether each data block is a duplicate data block through three - opportunity deduplication; The three - opportunity deduplication includes: D1: Search for the physical page PPN and logical page LPN corresponding to the fingerprint FP of the current data in the fingerprint index table. If the search is successful, then proceed to D2; otherwise, determine that the current data block is not a duplicate data block and the three - opportunity deduplication ends; D2: After reading the data corresponding to the PPN, compare it byte by byte with the current data block. If they are exactly the same, determine that the current data block is a duplicate data block and transfer to D3; otherwise, transfer to D5; D3: Query the logical page LPNnow corresponding to the current data block IO request in the address mapping table to obtain the corresponding physical page PPNnow. If the query is successful, decrement the reference count corresponding to PPNnow in the reference count table and transfer to D4; otherwise, directly transfer to D4; D4: Update the physical page corresponding to LPNnow in the address mapping table to PPN, insert the "PPN, LPNnow" entry in the address reverse mapping table, and increment the reference count corresponding to PPN in the reference count table. The three - chance duplicate data deletion ends; D5: Traverse and query in the fingerprint update cache in the order from new to old to check if the FP exists. If it exists, transfer to D6; otherwise, transfer to D7; D6: Obtain the physical page newPPN corresponding to the FP in the fingerprint update cache, read the data in newPPN, and compare it byte by byte with the current data block. If they are exactly the same, determine that the current data block is a duplicate data block and transfer to D7; otherwise, transfer to D9; D7: Query the physical page PPNnow corresponding to LPNnow in the address mapping table. If the query is successful, decrement the reference count corresponding to PPNnow in the reference count table and transfer to D8; otherwise, directly transfer to D8; D8: Update the physical page corresponding to LPNnow in the address mapping table to newPPN, insert the "newPPN, LPNnow" entry in the address reverse mapping table, increment the reference count corresponding to newPPN in the reference count table, and update the physical page corresponding to the FP in the fingerprint index table to newPPN. The three - chance duplicate data deletion ends; D9: Query the physical page latentPPN corresponding to LPN in the address mapping table. If the search is successful, transfer to D10; otherwise, determine that the current data block is not a duplicate data block and the three - chance duplicate data deletion ends; D10: Read the data corresponding to latentPPN and compare it byte by byte with the current data block. If they are exactly the same, determine that the current data block is a duplicate data block and transfer to D11; otherwise, determine that the current data block is not a duplicate data block and the three - chance duplicate data deletion ends; D11: Query the physical page corresponding to LPNnow in the address mapping table. If the query is successful, decrement the reference count corresponding to this physical page in the reference count table and transfer to D12; otherwise, directly transfer to D12; D12: Update the physical page corresponding to LPNnow in the address mapping table to latentPPN, insert the "latentPPN, LPNnow" entry in the address reverse mapping table, increment the reference count corresponding to latentPPN in the reference count table, and update the physical page corresponding to the FP in the fingerprint index table to latentPPN. The three - chance duplicate data deletion ends.

7. A computer program product, characterized in that, including a computer program; when the computer program is executed by a processor, implementing the deduplication method based on ZNS SSD according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, including a stored computer program, when the computer program is executed by a processor, controlling the device where the computer-readable storage medium is located to execute the deduplication method based on ZNS SSD according to any one of claims 1 to 6.

9. A deduplication controller based on ZNS SSD, characterized in that, including: a computer-readable storage medium for storing a computer program; and a processor for reading the computer program stored in the computer-readable storage medium and executing the deduplication method based on ZNS SSD according to any one of claims 1 to 6.

10. A storage system, characterized in that, including: a ZNS SSD, and the deduplication controller based on ZNS SSD according to claim 9.

Images