Persistent memory-based deduplication metadata management method

Abstract

The application discloses a kind of based on persistent memory's repeated data deletion metadata management method, including in DRAM management global fingerprint table and logical address to fingerprint mapping cache, in persistent memory management append redo log and global mapping table.Write path will first write metadata update into DRAM and form transaction record, sequentially append to persistent memory log using group commit mode, and with persistent memory read-write granularity size alignment persistence, reduce random write and lock contention;Checkpoint phase is according to the redundancy rate and the proportion of overwrite obtained by online load detection, dynamically select aggregation strategy, so as to significantly reduce metadata write amplification under the premise of ensuring crash consistency.Meanwhile, the application provides three kinds of adaptive deduplication mode: in low redundancy stage, by bypassing fingerprint management, in moderate redundancy stage, sample estimate redundancy, in high redundancy stage, enable full deduplication, so as to stably reduce metadata management cost and improve system throughput under different loads.

Description

Technical Field

[0001] This invention relates to the field of computer storage technology, and in particular to a method for deduplication and metadata management based on persistent memory. Background Technology

[0002] Among related technologies, deduplication technology identifies and eliminates redundant data blocks in storage systems, reducing solid-state drive (SSD) capacity usage and write bandwidth consumption. Therefore, it is widely used in backup archiving, virtualization images, container distribution, and storage systems handling massive amounts of data. Compared to ordinary block storage, deduplication systems, in addition to data writing, must also manage the mapping from logical addresses to fingerprints, and the mapping from fingerprints to physical locations and reference relationships, to support deduplication identification, physical block reuse, and reference counting management. As storage media performance continues to improve, system performance bottlenecks are gradually shifting from reading and writing the data ontology to the metadata path: fingerprint calculation and index lookup, mapping updates, reference relationship management, and related consistency control often become key factors determining throughput and tail latency.

[0003] Under real-world workloads, deduplication metadata management faces two more prominent challenges: First, the fine-grained, random, and frequent nature of updates. Write requests are typically split into fixed block sizes, meaning each block may trigger a mapping update. As the proportion of overwrite writes increases, the same logical address is repeatedly updated, accompanied by adjustments to old data reference relationships, making metadata updates more intensive. Second, the additional burden of crash consistency requirements. To ensure the system can recover to a consistent state after a power outage or crash, metadata updates often need to use logs or transaction mechanisms to record changes and replay them during recovery, introducing additional write path persistence overhead. If persistence is too frequent, the system will experience significant write amplification and latency jitter; if persistence is not timely enough, it will lead to increased log growth and recovery replay volume, thus lengthening recovery time and affecting system availability. Furthermore, when the checkpoint mechanism lacks effective merging capabilities, multiple updates to the same logical address within a short period, and multiple reference changes to the same fingerprint within the checkpoint cycle, will trigger a large number of duplicate writes to disk and duplicate replays, creating dual redundancy overhead during runtime and recovery. Summary of the Invention

[0004] This invention aims to at least partially solve one of the technical problems in the aforementioned technologies. To this end, one objective of this invention is to propose a metadata management method for deduplication based on persistent memory. This method first writes metadata updates into memory to form transaction records, then sequentially appends them to the persistent memory log using a group commit approach. Persistence is aligned with the granularity of persistent memory reads and writes, reducing random writes and lock contention. This results in a stable reduction in metadata management costs and improved system throughput under different loads.

[0005] To achieve the above objectives, this invention proposes a deduplication metadata management method based on persistent memory, comprising the following steps: initializing the metadata structure in memory, the metadata region in persistent memory, and consistency control information; receiving write request data blocks and calling a workload analysis module to obtain the deduplication mode, wherein the deduplication mode includes full deduplication, sampled deduplication, and direct write bypass; if the deduplication mode is direct write bypass mode, a new physical location is directly allocated to the data block and written to the solid-state drive, a direct write identifier for expressing the mapping relationship is generated and written to the transaction record, the old mapping of the logical address is read and reference maintenance or reclamation is performed in the case of overwrite write, the new mapping is updated to the mapping cache from logical address to fingerprint, and finally the transaction record is written to the thread-local buffer; if the deduplication mode is full deduplication or sampled deduplication mode, the data block fingerprint is calculated and the fingerprint index table is queried; when it is determined to be a duplicate data block, it is reused. The system has an existing physical location and updates the reference count. When a new data block is identified, a new physical location is allocated and written to the solid-state drive. Simultaneously, the fingerprint index table is updated, the old mapping of the logical address is read, and reference maintenance or reclamation marking is performed in the case of overwrite write. The new mapping is updated to the mapping cache from logical address to fingerprint. Finally, the transaction record is written to the thread-local buffer. When the thread-local buffer meets the commit condition, a batch of transaction records is sequentially appended to the transaction redo log in persistent memory in a group commit manner. After persistence is completed, the persistence progress pointer in the superblock is updated. This improves the overall throughput and efficiency of the system.

[0006] In addition, the metadata management method for deduplication based on persistent memory proposed in the above embodiments of the present invention may also have the following additional technical features:

[0007] Optionally, the initialization of the metadata structure in memory includes establishing a fingerprint index table, establishing a mapping cache from logical addresses to fingerprints, distinguishing between dirty data and clean data that need to be committed, allocating a thread-local buffer for the writing thread, and starting a workload analysis module to output the deduplication mode and commit strategy; the initialization of the metadata area in persistent memory includes creating or opening a persistent memory space and establishing a superblock; dividing and initializing the logical mapping table, fingerprint mapping table, and transaction redo log area, and completing persistence; the initialization of consistency control information includes setting log boundaries, persistence progress, and checkpoint position pointers to ensure that the system can be restored to a consistent state after a crash, initializing or opening the data area on the solid-state drive to store actual data blocks, and entering a normal service state.

[0008] Optionally, the process also includes a data read request step: receiving a read request data block, querying the cache of logical addresses to fingerprints to obtain the mapping relationship between logical addresses and fingerprints / identifiers; if no match is found, querying the logical mapping table in persistent memory; depending on the type of the obtained identifier, if it is a write-through identifier, directly parsing to obtain the physical location; if it is a fingerprint identifier, further querying the fingerprint index table to obtain the physical location; if the fingerprint index table is not found, querying from the logical mapping table and backfilling the fingerprint index table according to a strategy; reading data from the solid-state drive pointed to by the physical address and returning it; optionally, backfilling the hotspot mapping to the cache of logical addresses to fingerprints to improve subsequent access performance.

[0009] Optionally, the process also includes a checkpoint step: entering the checkpoint process when the checkpoint triggering conditions are met, the triggering conditions including dirty data mapping accumulating to a threshold, insufficient log space, or reaching a preset period; entering the checkpoint critical section and converging concurrent writes; writing uncommitted transactions into the transaction redo log and completing a commit, thus advancing the persistence progress to the latest consistency position; determining the scope of this checkpoint processing as from the previous checkpoint position to the latest persistence position; aggregating and organizing the transaction records within the checkpoint scope, writing the updates from logical addresses to fingerprints into the persistent logical mapping table in a merged manner; aggregating and organizing the fingerprint updates involved within the checkpoint scope, writing the fingerprint-to-physical location and reference count information into the persistent fingerprint mapping table, and writing the reclamation information; advancing the checkpoint position and persisting the superblock; cleaning up dirty data mappings and reclaiming log space according to a strategy.

[0010] Optionally, it also includes crash recovery steps: after the system crashes and restarts, read the superblock to obtain the checkpoint location, persistence progress, and log boundaries; identify the log range of the last valid commit to avoid replaying data that was not fully committed; load the persistent logical mapping table and the persistent fingerprint mapping table, and rebuild the index and runtime structure in memory; replay the committed transaction redo log records from the checkpoint location, restore the latest logical mapping and fingerprint reference relationship, and restore the reference management results under the overwrite scenario; after recovery is complete, update the runtime pointer and enter normal service.

[0011] In summary, by transforming high-frequency metadata operations from fine-grained random persistence to sequential append and batch merge processing, the number of concurrent contentions and persistence operations is reduced. This significantly reduces metadata update overhead and write amplification while ensuring crash consistency. It also supports checkpoint aggregation and fast recovery, shortens replay time after a crash, and can adaptively adjust deduplication strategies and commit strength according to workload changes, thereby improving the overall throughput and efficiency of the system. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating a persistent memory-based deduplication metadata management method according to an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of the overall architecture of a persistent memory-based deduplication metadata management device according to an embodiment of the present invention.

[0014] Figure 3 This is a schematic diagram of the metadata layout according to an embodiment of the present invention;

[0015] Figure 4 This is a schematic diagram of the online load detection process according to an embodiment of the present invention;

[0016] Figure 5 This is a schematic diagram of the incremental aggregation log processing flow according to an embodiment of the present invention;

[0017] Figure 6 This is a schematic diagram of the read / write request processing flow according to an embodiment of the present invention. Detailed Implementation

[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0019] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0020] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0021] Figure 1 This is a flowchart illustrating a persistent memory-based deduplication metadata management method according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0022] S101, initialize the metadata structure in memory, the metadata area in persistent memory, and the consistency control information.

[0023] As an example, initializing the metadata structure in memory includes establishing a fingerprint index table, establishing a mapping cache from logical addresses to fingerprints, distinguishing between dirty data and clean data that need to be committed, allocating thread-local buffers for write threads, and starting a workload analysis module to output deduplication mode and commit strategy; initializing the metadata area in persistent memory includes creating or opening persistent memory space and establishing a superblock; partitioning and initializing the logical mapping table, fingerprint mapping table, and transaction redo log area, and completing persistence; initializing consistency control information includes setting log boundaries, persistence progress, and checkpoint position pointers to ensure that the system can be restored to a consistent state after a crash; initializing or opening the data area on the solid-state drive to store actual data blocks; and entering a normal service state.

[0024] S102, receive write request data block, and call the workload analysis module to obtain the deduplication mode, wherein the deduplication mode includes full deduplication, sampling deduplication and direct write bypass.

[0025] S103, if the deduplication mode is the write-through bypass mode, then a new physical location is directly allocated for the data block and written to the solid-state drive, a write-through identifier for expressing the mapping relationship is generated and written to the transaction record, the old mapping of the logical address is read and reference maintenance or reclamation mark is executed in the case of overwrite write, the new mapping is updated to the mapping cache from logical address to fingerprint, and finally the transaction record is written to the thread local buffer.

[0026] S104. If the deduplication mode is full deduplication or sampled deduplication mode, calculate the data block fingerprint and query the fingerprint index table. When it is determined to be a duplicate data block, reuse the existing physical location and update the reference count. When it is determined to be a new data block, allocate a new physical location and write it to the solid-state drive. At the same time, update the fingerprint index table, read the old mapping of the logical address and perform reference maintenance or reclamation mark in the case of overwrite write. Update the new mapping to the mapping cache from logical address to fingerprint. Finally, write the transaction record to the thread local buffer.

[0027] S105: When the thread-local buffer meets the commit condition, a batch of transaction records are sequentially appended to the transaction redo log in persistent memory in a group commit manner. After persistence is completed, the persistence progress pointer in the superblock is updated.

[0028] As one embodiment, the method also includes a data read request step: receiving a read request data block, querying the cache of logical addresses to fingerprints to obtain the mapping relationship between logical addresses and fingerprints / identifiers, and if no match is found, querying the logical mapping table in persistent memory; depending on the obtained identifier type, if it is a write-through identifier, directly parsing to obtain the physical location, and if it is a fingerprint identifier, further querying the fingerprint index table to obtain the physical location, and if the fingerprint index table is not found, querying from the logical mapping table and backfilling the fingerprint index table according to the strategy; reading data from the solid-state drive pointed to by the physical address and returning it, and backfilling the hotspot mapping to the cache of logical addresses to fingerprints to improve subsequent access performance.

[0029] As one embodiment, the process also includes a checkpoint step: when the checkpoint triggering conditions are met, the checkpoint process is initiated. The triggering conditions include dirty data mapping accumulating to a threshold, insufficient log space, or reaching a preset period; entering the checkpoint critical section and converging concurrent writes; writing uncommitted transactions into the transaction redo log and completing a commit, thus advancing the persistence progress to the latest consistency position; determining the scope of this checkpoint processing as from the previous checkpoint position to the latest persistence position; aggregating and organizing the transaction records within the checkpoint scope, and writing the updates from logical addresses to fingerprints into the persistent logical mapping table in a merged manner; aggregating and organizing the fingerprint updates involved within the checkpoint scope, and writing the fingerprint-to-physical location and reference count information into the persistent fingerprint mapping table, and writing reclamation information; advancing the checkpoint position and persisting the superblock; cleaning up dirty data mappings and reclaiming log space according to a strategy.

[0030] As one embodiment, it also includes crash recovery steps: after the system crashes and restarts, it reads the superblock to obtain the checkpoint location, persistence progress, and log boundaries; identifies the log range of the last valid commit to avoid replaying data that was not fully committed; loads the persistent logical mapping table and the persistent fingerprint mapping table, and rebuilds the index and runtime structure in memory; replays the committed transaction redo log records from the checkpoint location, restores the latest logical mapping and fingerprint reference relationship, and restores the reference management results under the overwrite scenario; after recovery, it updates the runtime pointer and enters normal service.

[0031] In summary, the metadata management method for deduplication based on persistent memory according to the embodiments of this application has the following beneficial effects:

[0032] (1) Metadata updates in the write path are converted into append-only log records and persisted using group commit, which significantly reduces write amplification caused by fine-grained random writes and frequent persistence, thereby improving throughput under high-concurrency writes and reducing tail latency jitter; (2) Logical mapping and fingerprint mapping are aggregated and organized by different keys during the checkpoint phase, reducing repeated disk writes and repeated updates, reducing checkpoint write volume and effectively compressing the log size required for crash recovery, thereby shortening recovery time; (3) The deduplication mode and commit / aggregation strategy are adaptively adjusted through online workload analysis, enabling the system to reduce fingerprint maintenance overhead in the low redundancy phase, obtain better space benefits and control recovery costs in the high redundancy phase, thereby maintaining more stable overall efficiency under load changes.

[0033] To better understand the above technical solution, a specific embodiment will be provided for detailed explanation. The method includes the following steps:

[0034] 1) Initialization steps:

[0035] (1) Initialize the DRAM (memory) metadata structure, including: establishing a global fingerprint table (GFPT) to manage the correspondence between fingerprint FP and physical block address PBA, reference count ref, and quick digest quick_sig; establishing a mapping cache from logical block address to fingerprint (dcL2FP) and dividing it into a dirty entry area and a net entry area, wherein entries in the dirty entry area are fixedly retained until the checkpoint commit is completed, and entries in the net entry area are allowed to be eliminated according to the policy; initializing a transaction record batch buffer (TxnBatchBuffer) for each write thread; initializing the online load probing module (WorkloadProfiler) to maintain the sampling window and output the deduplication mode, group commit window parameter wal_flush_records and checkpoint aggregation flag, and then proceed to process (2).

[0036] (2) Initialize the persistent memory (PM) metadata structure, including: creating or opening persistent memory mapping regions and establishing superblocks; dividing and binding the following in persistent memory: persistent mapping base table from logical block address to fingerprint (L2FP), persistent mapping base table from fingerprint to physical block address and base reference count (FP2P), and transaction redo log (AtomLog); completing the initialization of L2FP, FP2P and AtomLog regions and performing necessary persistence, then proceed to process (3);

[0037] (3) Initialize log boundaries and checkpoint positions, including: setting checkpoint_lsn, persistent boundary_lsn, log tail pointer_lsn and AtomLog head / tail boundaries in the superblock; and setting the alignment granularity and flushing strategy of log records so that logs before durable_lsn can be reliably replayed after system crash, and then proceed to process (4).

[0038] (4) Open or create a solid-state drive (SSD) data area for storing unique data blocks, configure the capacity parameters and initialize the space management structure of the data area, and establish a record of the free space required for physical block allocation and reclamation; after initialization, bind the data area handle to the metadata management module, so that the system enters the normal read and write service state, complete the initialization, and the operation ends.

[0039] 2) Steps to write a request:

[0040] (1) Receive write request data block, call the online load detection module to output the deduplication mode of this write, the deduplication modes include: FULL (full deduplication), SAMPLE (sample deduplication) and DIRECT (direct write bypass); when this data block is determined to enter the DIRECT path or the non-sampled sub-path of SAMPLE, enter the DIRECT fast path and go to step (2); when this data block is determined to enter the FULL path or the sampled sub-path of SAMPLE, enter the fingerprint deduplication path and go to step (3).

[0041] (2) Allocate a new physical block address PBA for the logical block and write the data to the SSD; generate a direct write identifier direct_fp to express the recoverable mapping information from "logical block address LBA to physical block address PBA", and write direct_fp and PBA into the transaction record; obtain the old mapping old_fp of the LBA (first check dcL2FP, if not found, check the persistent L2FP base table); when it is an overwrite write and old_fp and direct_fp are different, perform reference maintenance or reclamation marking according to the type of old_fp: if old_fp is a FULL fingerprint, decrement its reference count ref in GFPT; if old_fp is a DIRECT direct write identifier, mark the old PBA as reclaimable according to its encoding information; write the new mapping "LBA→direct_fp" to dcL2FP. Dirty entry area; Construct transaction record TxnRecord, which at least includes lba, new_fp, old_fp, pba, quick_sig, pattern flag and aggregation flag; Write TxnRecord to thread local TxnBatchBuffer, and go to step (4);

[0042] (3) Calculate the data block fingerprint FP and quick_sig; search for FP in GFPT; when the search is successful and quick_sig is consistent, it is determined to be a duplicate block, the PBA of the successful entry is reused and its reference count ref is incremented; when there is no successful search or quick_sig is inconsistent, it is determined to be a new block, a new PBA is allocated and written to SSD, and at the same time, FP entries are inserted or updated in GFPT to record PBA, ref and quick_sig; obtain the old mapping old_fp of the LBA (first check dcL2FP, if no successful search, check the persistent L2FP base table); when it is an overwrite and old_fp is different from FP, perform reference maintenance or reclamation marking on old_fp: if old_fp is a FULL fingerprint, decrement its reference count ref in GFPT, and mark the corresponding physical block as reclaimable when ref reaches zero; if old_fp If the DIRECT class is a direct write identifier, then the old PBA is marked as recyclable according to its encoding information; the new mapping of LBA→FP is written to the dcL2FP dirty entry area; a transaction record TxnRecord is constructed; TxnRecord is written to the thread-local TxnBatchBuffer, and then proceed to step (4).

[0043] (4) When TxnBatchBuffer does not meet the commit condition, it continues to accumulate; when TxnBatchBuffer is full, reaches wal_flush_records, or triggers the synchronous persistence condition, a batch of TxnRecords is written to AtomLog in append mode, and a commit marker is written to the end of the batch; persistent flushing is performed on the append interval according to the predetermined alignment granularity; after flushing is completed, the durable_lsn and tail_lsn in the superblock are updated so that the log batches before durable_lsn can be reliably replayed after the system crashes; the write request processing ends.

[0044] 3) Read request processing steps:

[0045] (1) Upon receiving a read request data block, first query the fp corresponding to the LBA in dcL2FP; if no match is found, query the fp from the persistent L2FP base table and proceed to step (2).

[0046] (2) When fp is a DIRECT type direct write identifier, PBA is parsed from fp; when fp is a FULL type fingerprint, PBA and quick_sig are obtained by querying the fingerprint in GFPT; when GFPT is not hit, PBA is obtained by querying the persistent FP2P base table, and GFPT is backfilled according to the strategy, and then proceed to step (3).

[0047] (3) Read data from the SSD pointed to by PBA and return it; when it is a FULL fingerprint and verification is enabled, perform consistency verification on the read data based on quick_sig or fingerprint; fill the hot spot mapping back into the dcL2FP net entry area and update the hot spot information of GFPT; the read request processing ends.

[0048] 4) Checkpoint steps:

[0049] (1) When the number of dirty entries in dcL2FP exceeds the threshold, the available space in AtomLog is insufficient, or the periodic checkpoint condition is reached, a checkpoint is triggered; the checkpoint state is set and the checkpoint critical section is entered, the concurrent write thread is paused or blocked to converge the concurrent update, and then proceed to step (2).

[0050] (2) Append all uncommitted TxnRecords in the TxnBatchBuffer of all threads to AtomLog and write the corresponding commit_marker; perform persistent flushing on the appended log range to advance durable_lsn to the latest stable position; read checkpoint_lsn and durable_lsn from the superblock and determine the scope of this checkpoint processing as [checkpoint_lsn, durable_lsn] (this scope only includes transaction records that have been committed and persisted), and go to step (3);

[0051] (3) Log scanning and L2FP aggregation write-back steps: sequentially scan the TxnRecord within the range; merge multiple updates of the same LBA, and only keep the latest new_fp of the LBA; write the merged L2FP updates to the persistent L2FP base table in batches, and perform persistence at the batch boundary; retain version number or timestamp information during the write-back process to ensure that "new value overwrites old value", and go to step (4);

[0052] (4) Collect the set of FULL fingerprints reached by this checkpoint based on the checkpoint aggregation flag, and remove duplicates from the set; perform FP2P base table update only once for each fingerprint, and the update content is based on the authoritative PBA and ref recorded in GFPT; when ref is zero or there is a recycling flag, write the corresponding physical block recycling information into FP2P or recycling queue, and go to step (5).

[0053] (5) Push checkpoint_lsn to durable_lsn and persist the superblock so that the push result is visible after the system crashes; clear the dcL2FP dirty entry area to release fixed memory and retain hot spot mapping in the clean entry area according to the policy; perform truncation or space reclamation on AtomLog based on the pushed checkpoint_lsn to release log space; checkpoint ends.

[0054] 5) Crash recovery steps:

[0055] (1) After the system crashes and restarts, read the superblock to obtain checkpoint_lsn, durable_lsn, tail_lsn and the head / tail boundary of AtomLog; identify the last valid commit boundary based on the commit_marker, limit the upper bound of the recovery replay to the durable_lsn that has been committed and persisted, avoid replaying half-written or uncommitted TxnRecord, and proceed to step (2).

[0056] (2) Reconstruct the base entries of GFPT from the persistent FP2P base table or load fingerprint entries on demand; load the base LBA→fp mapping view from the persistent L2FP base table as the initial state for log replay, and proceed to step (3).

[0057] (3) Scan AtomLog sequentially from checkpoint_lsn to durable_lsn, read TxnRecord one by one and perform idempotent replay: update LBA mapping to new_fp of TxnRecord; when it is an overwrite write and old_fp is different from new_fp, perform reference decrement or recycling mark recovery on old_fp, and perform reference increment or recovery to the correct value on new_fp; when TxnRecord is marked as DIRECT, restore LBA→PBA relationship according to the direct_fp and PBA information recorded therein; when TxnRecord is marked as FULL, correct GFPT entries according to FP and quick_sig recorded therein and ensure that ref is consistent with PBA, and go to step (4);

[0058] (4) Runtime recovery steps: Fill the latest mapping obtained from the replay back into the dcL2FP dirty entry area or clean entry area, and clean up inconsistent entries; restore the log tail pointer to tail_lsn and allow subsequent writes to continue appending AtomLog; persist the recovery completion flag or update the superblock version information if necessary; the crash recovery ends.

[0059] Figure 2 This is a schematic diagram of the overall architecture of the device of the present invention. The device is divided into three parts: a memory-side metadata layer, a persistent memory metadata layer, and a solid-state drive (SSD) data layer. The memory-side metadata layer is used to carry fast paths for writing and reading, mainly including a global fingerprint index structure and a logical address mapping cache, and can provide transaction buffers for concurrent threads to reduce lock contention. The persistent memory metadata layer is used to provide crash consistency guarantees and persistent storage, mainly including a superblock, a transaction redo log, and a persistent base table for logical mapping and fingerprint mapping. The superblock is used to record log boundaries and checkpoint positions, the transaction redo log is used to record metadata change trajectories, and the persistent base table is used to save the stable state after the checkpoint. The SSD data layer is used to store actual data blocks, only writing to new data blocks that are determined to be unique, and providing physical block read services to the read path, thereby forming a layered architecture of accelerated memory access, persistent memory guaranteeing consistency, and SSD storage of unique block data.

[0060] Figure 3This is a schematic diagram of the metadata layout of the present invention; metadata is functionally divided and stored in different media. The memory includes a fingerprint index table and a logical mapping cache, where the logical mapping cache is further divided into a dirty entry area and a clean entry area. Dirty entries are used to temporarily store the latest mapping updates that have not yet been committed at checkpoints, while clean entries are used to cache hot mappings to improve the hit rate. The persistent memory includes a superblock area, a redo log area, a logical mapping persistent base table area, and a fingerprint mapping persistent base table area. The superblock area records checkpoint locations and persistent log boundaries. The redo log area sequentially appends transaction records generated by the write path. The logical mapping base table area stores stable mapping relationships from logical addresses to fingerprints / identifiers, and the fingerprint mapping base table area stores information such as fingerprints to physical locations and reference relationships. Through this layout, frequent updates during writes are kept in memory and the sequential log as much as possible, while persistent base table updates are concentrated at the checkpoint stage, thereby reducing random write amplification.

[0061] Figure 4 The diagram illustrates the online load probing process of this invention, which involves the following operations: During operation, the system continuously monitors write traffic and statistically updates key indicators within a sliding window. These indicators include at least redundancy level and write coverage ratio. The system calculates the current load state based on the sampling statistics and outputs the corresponding deduplication mode and commit strategy parameters. The deduplication mode includes at least three types: full deduplication, sampling deduplication, and direct write bypass, used to balance fingerprint maintenance overhead and space gains under different redundancy scenarios. The commit strategy parameters adjust the group commit batch size and checkpoint aggregation intensity to control recovery and replay costs during high-coverage writes or high-redundancy phases, and reduce index maintenance and persistence pressure during low-redundancy phases. The mode and parameters output by the online load probing module are read by the write path and checkpoint path, thereby achieving adaptive adjustment according to load changes.

[0062] Figure 5 The diagram illustrates the incremental aggregation log processing flow of this invention. The following operations are performed: When the log growth or dirty mapping accumulation reaches a threshold, the system triggers a checkpoint, extracts incremental updates within the checkpoint range from the recorded transaction log, merges multiple updates to the same logical address, and retains only the latest mapping result; aggregates multiple accesses to the same fingerprint, and performs only one persistent update, thereby transforming the random write pressure of a large number of fine-grained updates into sequential log appending and batch merging to disk, reducing write amplification and reducing recovery replay volume.

[0063] Figure 6The flowchart of the read / write request processing of this invention is shown below. The following operations are performed: Upon system startup, the initialization module is fully configured: a Global Fingerprint Table (GFPT) is established in DRAM to manage the mapping relationship between fingerprints and physical addresses; a logical address-to-fingerprint cache (dcL2FP) is created and dirty / clean regions are divided; a transaction buffer is allocated for each write thread; in persistent memory (PM), logical mapping table (L2FP), fingerprint mapping table (FP2P), and transaction redo log (AtomLog) regions are divided, and a superblock record key pointer is established; finally, the SSD data area is initialized to complete storage space preparation. After the system enters the running state, the write request processing receives application write requests and dynamically selects one of three deduplication modes through the load analyzer: for low-redundancy loads, the direct write bypass (DIRECT) mode is used, directly allocating physical addresses to write data; for medium-redundancy loads, the sampled deduplication (SAMPLE) mode is used, selectively calculating fingerprints for deduplication; for high-redundancy loads, the full deduplication (FULL) mode is enabled, fully calculating fingerprints and querying the GFPT table. Regardless of the chosen path, the system constructs metadata updates as transaction records, caches them in a thread-local buffer, and sequentially appends them to the AtomLog in the PM (Persistent Memory) in a group commit manner when the commit conditions are met. After persistence is complete, the persistence progress pointer in the superblock is updated. Read request processing adopts a cache-first strategy. First, it queries the dcL2FP cache to obtain the mapping relationship. If it misses, it queries the persistent L2FP base table in the PM. Depending on the obtained identifier type, if it is a write-through identifier, it directly resolves the physical address; if it is a fingerprint identifier, it further queries the GFPT or FP2P base table to obtain the physical location, and finally reads the data from the SSD and returns it. As the system runs, the checkpoint processing module will be triggered periodically or when conditions are met: when dirty mappings accumulate to a threshold, log space is insufficient, or a preset period is reached, the system enters the checkpoint critical section, suspends concurrent writes, flushes all uncommitted transactions to the log, and then performs aggregation processing on the log records within the checkpoint range—merging and updating logical mappings by address, deduplicating and updating fingerprint mappings by fingerprint, and finally writing the aggregation results back to the persistent base table in batches, updating the checkpoint pointer, and reclaiming log space. When the system encounters an unexpected crash, the crash recovery module ensures data consistency: after restarting, it first reads the superblock to obtain the key pointer, loads the persistent base table to rebuild the initial state, then replays the committed records in the AtomLog log starting from the checkpoint position, restores the latest mapping relationship through idempotent operations, and finally restores the system to the last consistent state before the crash, and re-enters normal service.

[0064] Through the above embodiments, this application prioritizes storing frequently accessed location information and relationships on fast media or memory during system operation. Local maintenance and hotspot backfilling reduce random searches of the underlying persistent media. For unavoidable persistent access, discrete queries are transformed into sequential, batch-processable access patterns, such as appending records to aggregate updates and batch merging to centrally persist data to disk, thereby reducing the proportion of random read / write operations and access amplification. Furthermore, after a crash or abnormal interruption, the system restores to a consistent state by recording change trajectories or checkpoint states to prevent metadata inconsistencies due to partial update loss. During the recovery phase, the amount of replay and recovery time are controlled to balance runtime performance and availability.

[0065] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0066] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0067] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0068] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0069] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0070] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0071] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0072] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0073] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0074] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0075] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0076] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for deduplication and metadata management based on persistent memory, characterized in that, Includes the following steps: Initialize the metadata structure in memory, the metadata region in persistent memory, and the consistency control information, including log boundaries, persistence progress, and checkpoint position pointers; Receive write request data blocks and call the workload analysis module to obtain the deduplication mode, wherein the deduplication mode includes full deduplication, sampled deduplication and direct write bypass; If the deduplication mode is the write-through bypass mode, then a new physical location is directly allocated for the data block and written to the solid-state drive. A write-through identifier is generated to express the mapping relationship and written to the transaction record. The old mapping of the logical address is read and reference maintenance or reclamation mark is executed in the case of overwrite write. The new mapping is updated to the mapping cache from logical address to fingerprint. Finally, the transaction record is written to the thread local buffer. If the deduplication mode is full deduplication or sampled deduplication, the data block fingerprint is calculated and the fingerprint index table is queried. When it is determined to be a duplicate data block, the existing physical location is reused and the reference count is updated. When it is determined to be a new data block, a new physical location is allocated and written to the solid-state drive. At the same time, the fingerprint index table is updated, the old mapping of the logical address is read, and in the case of overwrite, reference maintenance or reclamation mark is performed. The new mapping is updated to the mapping cache from logical address to fingerprint. Finally, the transaction record is written to the thread local buffer. When the thread-local buffer meets the commit condition, a batch of transaction records are sequentially appended to the transaction redo log in persistent memory in a group commit manner. After persistence is completed, the persistence progress pointer in the superblock is updated.

2. The method for deduplication and metadata management based on persistent memory as described in claim 1, characterized in that, Initializing the metadata structure in memory includes creating a fingerprint index table, creating a mapping cache from logical addresses to fingerprints, distinguishing between dirty data and clean data that need to be committed, allocating a thread-local buffer for the writing thread, and starting the workload analysis module to output the deduplication mode and commit strategy. Initializing the metadata area in persistent memory includes creating or opening persistent memory space and establishing a superblock; Divide and initialize the logical mapping table, fingerprint mapping table, and transaction redo log area, and then persist them. Initializing consistency control information includes setting log boundaries, persistence progress, and checkpoint position pointers to ensure recovery to a consistent state after a crash, initializing or opening the data area on the solid-state drive to store actual data blocks, and entering a normal service state.

3. The method for deduplication and metadata management based on persistent memory as described in claim 2, characterized in that, It also includes the data read request step: Receive read request data blocks, query the mapping cache from logical address to fingerprint to obtain the mapping relationship from logical address to fingerprint / identifier. If no match is found, query the logical mapping table in persistent memory. Depending on the type of the obtained identifier, if it is a direct write identifier, the physical location is directly parsed; if it is a fingerprint identifier, the fingerprint index table is further queried to obtain the physical location; if the fingerprint index table is not found, the logical mapping table is queried and the fingerprint index table is backfilled according to the strategy. Data is read from the solid-state drive pointed to by the physical address and returned. Hotspot mappings are then filled back into the logical address to fingerprint mapping cache to improve subsequent access performance.

4. The method for deduplication and metadata management based on persistent memory as described in claim 2, characterized in that, It also includes checkpoint steps: When the checkpoint triggering conditions are met, the checkpoint process is entered. The triggering conditions include dirty data mapping accumulating to a threshold, insufficient log space, or reaching a preset period; the checkpoint critical section is entered and converged for concurrent writing. Write the uncommitted transactions to the transaction redo log and complete a commit, advancing the persistence progress to the latest consistent position; determine the scope of this checkpoint to be from the last checkpoint position to the latest persistence position; Aggregate and organize transaction records within the checkpoint scope, and write the updates from logical addresses to fingerprints into the persistent logical mapping table in a merged manner; The fingerprint updates within the checkpoint area are aggregated and organized, and the fingerprint-to-physical-location and reference count information are written into the persistent fingerprint mapping table, along with the recycling information. Advance the checkpoint location and persist the superblock; clean up dirty data mappings and reclaim log space according to the policy.

5. The method for deduplication and metadata management based on persistent memory as described in claim 2, characterized in that, It also includes crash recovery steps: After a system crash and restart, the superblock is read to obtain the checkpoint location, persistence progress, and log boundaries; the log range of the last valid commit is identified to avoid replaying data that was not fully committed. Load the persistent logical mapping table and persistent fingerprint mapping table, and reconstruct the index and runtime structure in memory; Starting from the checkpoint, replay the committed transaction redo log records, restore the latest logical mapping and fingerprint reference relationship, and restore the reference management results in the case of overwrite; After the recovery is complete, update the running pointer and resume normal service.

Images