Data deduplication method, device, storage medium, electronic device and program product

Abstract

The present application discloses a data deduplication method, device, storage medium, electronic device and program product, which relate to the field of data storage technology. The method includes setting a memory layer of a tree-like storage architecture in the memory, and online deduplication only requires memory read operations without accessing the hard disk, avoiding the delay and overhead of hard disk read operations in related technologies, and can quickly respond to data writes, significantly improving data processing efficiency; background deduplication is adopted in multiple hard disk layers, and the corresponding deduplication strategy is automatically triggered after the corresponding trigger strategy is met, thereby reducing frequent online hard disk reading and writing, reducing the disk reading frequency, and improving the overall performance of the all-flash storage array; wherein, different deduplication strategies are adopted for the memory layer, the first hard disk layer and other hard disk layers, and differentiated deduplication strategies are formulated according to the characteristics of different levels to make the deduplication operation more targeted and efficient, which can improve efficiency and accuracy, and solve the problem of large disk reading overhead in online data deduplication in related technologies.

Description

Technical Field

[0001] The present application relates to the field of data storage technology, and in particular to a data deduplication method, device, storage medium, electronic device, and program product. Background Art

[0002] All-flash storage arrays, with their exceptional high performance and low latency, have occupied a significant position in the enterprise storage market. As data volumes continue to rise, duplicate data not only wastes valuable storage space but also increases the complexity and cost of data backup, recovery, and long-term management.

[0003] Related deduplication technologies typically rely on a fingerprint database to identify and remove redundant data. This database is often implemented as a log file or B+ tree. During deduplication, the system traverses the entire fingerprint database, comparing the fingerprint of each new data block with the existing fingerprints in the database to identify and remove duplicate data.

[0004] However, related data deduplication technologies have obvious limitations, especially in global deduplication scenarios. Each data deduplication operation requires reading the entire fingerprint database from beginning to end, and the entire database needs to be rewritten after the operation is completed, resulting in a large amount of read and write overhead, which seriously affects the overall performance of the storage system. Summary of the Invention

[0005] The present application provides a data deduplication method, device, storage medium, electronic device and program product to at least solve the problem of high disk reading overhead in online data deduplication in related technologies.

[0006] The present application provides a data deduplication method, comprising: before writing a data block to be written into a memory layer of a tree-like storage architecture, performing an online deduplication operation on the memory layer according to a first deduplication strategy; the tree-like storage architecture includes multiple storage layers; the memory layer refers to the first storage layer among the multiple storage layers, which is used to accelerate the writing of the data block to be written and support online deduplication; the multiple other storage layers among the multiple storage layers except the first storage layer are multiple hard disk layers, which are used for background global deduplication and long-term data storage; the memory layer and the multiple hard disk layers adopt different deduplication strategies; when the first hard disk layer in the tree-like storage architecture meets the first trigger strategy, performing background deduplication operation on the first hard disk layer according to the second deduplication strategy; when the other hard disk layers among the multiple hard disk layers except the first hard disk layer meet the second trigger strategy, performing background deduplication operation on the other hard disk layers according to the third deduplication strategy.

[0007] The present application also provides a data deduplication device, including: a first deduplication module, used to perform online deduplication operations on the memory layer of the tree-like storage architecture according to a first deduplication strategy before writing the data block to be written into the memory layer of the tree-like storage architecture; the tree-like storage architecture includes multiple storage layers; the memory layer refers to the first storage layer among the multiple storage layers, used to accelerate the writing of the data block to be written and support online deduplication; the multiple other storage layers among the multiple storage layers except the first storage layer are multiple hard disk layers, used for background global deduplication and long-term data storage; the memory layer and the multiple hard disk layers adopt different deduplication strategies; a second deduplication module, used to perform background deduplication operations on the first hard disk layer in the tree-like storage architecture according to a second deduplication strategy when the first hard disk layer meets the first trigger strategy; a third deduplication module, used to perform background deduplication operations on the other hard disk layers according to a third deduplication strategy when the other hard disk layers among the multiple hard disk layers except the first hard disk layer meet the second trigger strategy.

[0008] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data deduplication methods when executing the computer program.

[0009] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data deduplication methods are implemented.

[0010] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data deduplication methods when executed by a processor.

[0011] Through this application, the memory layer of the tree-like storage architecture is located in the memory, and online deduplication only requires memory read operations without accessing the hard disk, avoiding the delay and overhead of hard disk read operations in related technologies, and can quickly respond to data writes, significantly improving data processing efficiency; background deduplication is adopted in multiple hard disk layers, and the corresponding deduplication strategy is automatically triggered after the corresponding trigger strategy is met, reducing frequent online hard disk reads and writes, reducing the disk reading frequency, and improving the overall performance of the all-flash storage array; among them, the memory layer, the first hard disk layer and other hard disk layers adopt different deduplication strategies, and differentiated deduplication strategies are formulated according to the characteristics of different levels to make deduplication operations more targeted and efficient, which can improve efficiency and accuracy, and comprehensively identify and eliminate duplicate data at each level through layered deduplication, eliminate duplicate data, avoid duplicate storage, reduce storage space occupancy, improve space utilization, and reduce storage costs, solving the problem of large disk reading overhead in online data deduplication in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] Figure 1 An application scenario diagram of a data deduplication method provided in an embodiment of the present application.

[0014] Figure 2 A flowchart of a data deduplication method provided in an embodiment of the present application.

[0015] Figure 3 A schematic diagram of a tree-like storage architecture provided in an embodiment of the present application.

[0016] Figure 4 A flowchart of a memory layer deduplication method provided in an embodiment of the present application.

[0017] Figure 5 A structural diagram of a data deduplication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0020] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] According to one aspect of the embodiment of the present application, a data deduplication method is provided. Optionally, in this embodiment, the above data deduplication method can be applied to, but is not limited to, Figure 1The hardware environment shown includes a terminal device 102 and a server 104. The server 104 can be connected to the terminal device 102 via a network and can be used to provide services (e.g., application services, etc.) for the terminal device 102 or a client installed on the terminal device 102. A database can be set on the server 104 or independently of the server 104 to provide data storage services for the server 104.

[0022] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, or a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity) and Bluetooth. The terminal device 102 may be, but is not limited to, a personal computer (PC), a mobile phone, a tablet computer, etc. The server 104 may be, but is not limited to, a cloud server, a server cluster, or other server types.

[0023] The data deduplication method of the embodiment of the present application can be executed by the server 104, or by the terminal device 102, or jointly by the server 104 and the terminal device 102. The data deduplication method of the embodiment of the present application can also be executed by the client installed on the terminal device 102.

[0024] Taking the data deduplication method of this embodiment executed by the server 104 as an example, Figure 2 This is a flow chart of an optional data deduplication method according to an embodiment of the present application. Figure 2 As shown, the process of the method may include the following steps:

[0025] S202. Before writing the data block to be written into the memory layer of the tree-like storage architecture, perform online deduplication operation on the memory layer according to the first deduplication strategy; the tree-like storage architecture includes multiple storage layers; the memory layer refers to the first storage layer among the multiple storage layers, which is used to accelerate the writing of the data block to be written and support online deduplication; the multiple storage layers other than the first storage layer among the multiple storage layers are multiple hard disk layers, which are used for background global deduplication and long-term data storage; the memory layer and the multiple hard disk layers adopt different deduplication strategies.

[0026] S204: When the first hard disk layer in the tree-like storage architecture satisfies the first triggering strategy, perform background deduplication operation on the first hard disk layer according to the second deduplication strategy.

[0027] S206 . When the hard disk layers other than the first hard disk layer in the multiple hard disk layers satisfy the second triggering strategy, perform background deduplication operations on the other hard disk layers according to the third deduplication strategy.

[0028] Among them, the data deduplication method of this application belongs to the field of data storage technology, and can be applied to all-flash arrays and storage systems based on tree-like storage architecture, and is specifically applied to scenarios such as enterprise data center storage optimization, cloud storage services, and solid-state hard drive management.

[0029] The data deduplication method of the present application is specifically applied to the data deduplication scenario of an all-flash storage array. Specifically, the all-flash storage array forms a RAID (Redundant Array of Independent Disks) and a storage pool by combining multiple solid-state drives (SSDs), and further divides the logical unit numbers (LUNs) to provide to the host, thereby providing efficient and reliable block storage services for various applications. The tree-like storage architecture is a hierarchical data storage and indexing mechanism in an all-flash storage array. Its design aims to optimize the efficiency of data writing, reading, and management by distributing data and metadata across multiple levels. The tree-like storage architecture includes multiple storage layers, each of which can have different storage media and optimization strategies to adapt to its specific access patterns and performance requirements. For example, the tree-like storage architecture can adopt an LSM-Tree (Log-Structured Merge-Tree) structure, or other tree-like data structures that can support hierarchical storage and efficient queries.

[0030] The lower levels of the tree-like storage architecture tend to use persistent storage (such as hard disks) for long-term data storage, while the higher levels may use fast-access media (such as memory) to speed up data processing and response speed. Therefore, in an embodiment of the present application, the first storage layer among the multiple storage layers of the tree-like storage architecture is set as the memory layer (denoted as the L0 layer), and the multiple storage layers other than the first storage layer in the multiple storage layers are multiple hard disk layers (denoted as the L1 layer and below). The data is first stored in the memory layer, and gradually migrated to the hard disk layer asynchronously, and dynamically rearranged between different layers through merge operations. Among them, the memory layer refers to the initial storage layer in the data writing process, which is completely located in the memory and supports online deduplication. Among them, online deduplication refers to quickly responding to data write requests through memory hash comparison to ensure that new data blocks are not stored repeatedly with existing data in the memory. The disk layer refers to all other layers in the tree-like storage architecture except the memory layer. It resides on persistent storage devices such as SSDs (Solid State Drives) or HDDs (Hard Disk Drives). It is used for long-term data storage and global deduplication, optimizing data storage efficiency and query performance during the merge process through background deduplication. Global deduplication refers to the background deduplication process performed at the disk layer (L1 and below), involving cross-layer and cross-file data block fingerprint comparisons to eliminate all redundant data in the storage system.

[0031] In the embodiment of the present application, the data structure of the memory layer usually adopts a concurrency-optimized memory index structure (such as a lock-free skip list or a B+ tree), supports high-throughput writing and low-latency reading, and mainly uses MemTable as the data carrier. The hard disk layer adopts an immutable, sequential read-optimized file format. For example, the data stored in the hard disk layer adopts the SSTable (SortedString Table, referred to as sorted string table) format, and SSTable is a persistent storage format. In this embodiment, the tree-like storage architecture has received widespread attention in the field of data storage due to its efficient writing performance and hierarchical storage characteristics. The tree-like storage architecture effectively optimizes the writing performance and reduces the performance overhead caused by random writing by first writing the data to the MemTable in the memory, and then writing it sequentially in batches to the SSTable file on the hard disk, and performing merge operations regularly. In terms of managing metadata, the tree-like storage architecture has shown significant advantages and can provide an efficient writing and query mechanism. In particular, its performance advantages are more prominent when processing large-scale metadata.

[0032] Due to the hierarchical storage characteristics of the tree-like storage architecture, when performing fingerprint queries, it may be necessary to read data from multiple layers of SSTable files, resulting in frequent disk read operations and increased I / O burden. This is particularly evident in online deduplication scenarios, because online deduplication requires real-time processing of newly written data blocks, and frequent disk read operations will seriously affect the system's response speed and throughput. To address the above problems, this embodiment proposes a hierarchical deduplication method that combines online deduplication and background deduplication. This method adopts differentiated deduplication strategies for different layers of the tree-like storage architecture. For example, for the L0 layer (i.e., the MemTable in memory), since all its data is stored in memory, only memory read operations are required during online deduplication, without accessing the hard disk, thereby significantly improving the efficiency of online deduplication. For the L1 layer and lower layers (i.e., the SSTable files on the hard disk), background deduplication operations are performed during the background merge process. Through this hierarchical deduplication strategy, while maintaining the efficient write performance of the tree-like storage architecture, the disk read overhead during the online deduplication process is effectively reduced, thereby improving the overall performance and efficiency of the all-flash storage array.

[0033] In the embodiment of this application, Figure 3 A schematic diagram of a tree-like storage architecture provided in an embodiment of the present application is shown as follows: Figure 3As shown, the memory layer and multiple hard disk layers use different deduplication strategies. The first hard disk layer in the tree-like storage architecture uses a different deduplication strategy than the other hard disk layers. All hard disk layers except the first hard disk layer in the tree-like storage architecture use the same deduplication strategy: the memory layer uses the first deduplication strategy, the first hard disk layer uses the second deduplication strategy, and all hard disk layers except the first hard disk layer use the third deduplication strategy. In the tree-like storage architecture, all hard disk storage layers (such as L2 and L3) other than the memory layer (L0) that directly interacts with memory and the first hard disk layer (L1) are collectively referred to as "other hard disk layers." The first deduplication strategy is an online deduplication strategy. For example, online detection of newly written data is performed by comparing it with the baseline data after the most recent deduplication operation to reduce full hash lookups, which is suitable for data environments with low change rates. Alternatively, fingerprints are calculated only for the changed portion of a data block and incrementally compared with the existing fingerprints to reduce computational complexity, which is suitable for scenarios with frequent small modifications. The second deduplication strategy refers to the background deduplication method applied to the first disk layer (L1) of the tree-like storage architecture. For example, the second deduplication strategy can use data fingerprint compression technology at the L1 layer to reduce the storage space of fingerprint data, thereby speeding up fingerprint comparison and improving background deduplication efficiency. Alternatively, it can leverage the multi-threaded parallel processing capabilities of L1 layer SSTable files to simultaneously compare the fingerprints of multiple files, accelerating the background deduplication process and reducing processing time. The third deduplication strategy is applied to the background deduplication solution of deeper layers (L2+) in the tree-like storage architecture beyond the first disk layer (L1). For example, the third deduplication strategy can use a timestamp mechanism at deeper layers (L2 and below) to perform global deduplication periodically (e.g., weekly), centrally processing historical data and reducing the impact of real-time deduplication on performance. Alternatively, it can set a storage space utilization efficiency threshold. When the space utilization rate of disk layers below the L2 layer falls below a preset value, the third deduplication strategy is activated to reclaim space occupied by redundant data and improve overall storage efficiency.

[0034] Before writing a data block, the memory layer executes the first deduplication policy. The second deduplication policy is only executed if the first disk layer in the tree-like storage architecture meets the first trigger policy. The third deduplication policy is only executed if the second trigger policy is met for all disk layers other than the first. The first trigger policy refers to the conditions that trigger background deduplication for the first disk layer (i.e., L1) in the tree-like storage architecture. For example, the first trigger policy may include a threshold for the number of SSTable files, storage space usage, or a specific time interval. When any one or more of these conditions are met, the system initiates background deduplication, i.e., the second deduplication policy, to optimize storage efficiency and performance. The second trigger policy refers to the rules that trigger background deduplication for all disk layers (L2 and below) in the tree-like storage architecture other than the first disk layer. For example, the second trigger policy may be based on more complex evaluation criteria, such as overall storage efficiency or the aging of data at a specific layer. It aims to initiate global, accurate deduplication at the right time to ensure that long-term stored data is free of redundancy. In this embodiment, the first trigger strategy and the second trigger strategy can be to start the merge operation, that is, the second deduplication strategy and the third deduplication strategy are triggered during the merge process, and the merge cycle is set at the same time. For example, the first trigger strategy and the second trigger strategy can be size triggering, time triggering or mixed strategies. For example, the first trigger strategy and the second trigger strategy can specifically be to trigger a background merge when the number of SSTables in a certain layer reaches a threshold (such as L1 layer ≥ 4 files).

[0035] Optionally, the server's all-flash storage array first divides the tree-like storage architecture into layers and configures the storage parameters and deduplication strategies for each layer; when the data blocks to be written are written at a preset fixed granularity, the data blocks to be written are first cached in the MemTable and deduplication is performed according to the first deduplication strategy. The all-flash storage array monitors the status of the first hard disk layer and automatically triggers the second deduplication strategy when the first hard disk layer meets the first trigger strategy. The all-flash storage array monitors the other hard disk layers (L2 layer and below) in the tree-like storage architecture except the first hard disk layer and automatically triggers the third deduplication strategy when the other hard disk layers meet the second trigger strategy.

[0036] Through this embodiment, the memory layer of the tree-like storage architecture is located in the memory, and online deduplication only requires memory read operations without accessing the hard disk, avoiding the delay and overhead of hard disk read operations in related technologies, and can quickly respond to data writes, significantly improving data processing efficiency; background deduplication is adopted in multiple hard disk layers, and the corresponding deduplication strategy is automatically triggered after the corresponding trigger strategy is met, reducing frequent online hard disk reads and writes, reducing the disk reading frequency, and improving the overall performance of the all-flash storage array; among them, the memory layer, the first hard disk layer and other hard disk layers adopt different deduplication strategies, and differentiated deduplication strategies are formulated according to the characteristics of different levels to make deduplication operations more targeted and efficient, which can improve efficiency and accuracy, and comprehensively identify and eliminate duplicate data at each level through layered deduplication, eliminate duplicate data, avoid duplicate storage, reduce storage space occupancy, improve space utilization, reduce storage costs, and solve the problem of large disk reading overhead in online data deduplication in related technologies.

[0037] In an exemplary embodiment, the memory layer stores a hash table; the hash table stores fingerprints of data blocks stored in the memory layer and pointers to physical addresses of the data blocks.

[0038] The hash table is the core data structure of the memory layer, storing the mapping between data block fingerprints and their corresponding physical addresses. Using a hash function, the hash table converts fingerprints into indexes, enabling rapid location and duplication checking of data blocks. This allows for extremely fast deduplication decisions, avoiding duplicate data writes and reducing storage space waste.

[0039] A data block's fingerprint typically refers to a fixed-length hash value (such as SHA-256 or xxHash64) calculated using a hash algorithm that represents the data block's contents. In a storage system, when a data block is written to a physical storage medium (such as an SSD or HDD), it occupies a certain amount of physical space. The starting position of this space is considered the data block's physical address. A pointer to a data block's physical address is actually a metadata structure that records the mapping between the data block's logical address and its actual physical address. During data deduplication, if a new data block is detected to have the same content as an already stored data block, the system will not write it again. Instead, it will update the logical address mapping to point to the physical address of the existing data block. This logical address update is essentially achieved through pointer redirection. This pointer mechanism can effectively reduce storage space usage and improve storage efficiency.

[0040] In some embodiments, before writing the data block to be written into the memory layer of the tree-like storage architecture, performing an online deduplication operation on the memory layer according to a first deduplication strategy includes:

[0041] Before writing the data block to be written into the memory layer, the fingerprint of the data block to be written is calculated and the hash table is queried. If the hash table contains a target fingerprint that is the same as the fingerprint of the data block to be written, the pointer in the hash table pointing to the physical address of the data block corresponding to the target fingerprint is updated.

[0042] The target fingerprint is the fingerprint found in the hash table that directly matches the hash fingerprint of the data block to be written. If the target fingerprint found in the hash table matches the fingerprint of the data block to be written, it indicates that the data block is duplicated. In this case, the system will update the pointer corresponding to the target fingerprint to point to the physical address of the original data block.

[0043] Optionally, Figure 4 A flowchart of a memory layer deduplication method provided in an embodiment of the present application is shown as follows: Figure 4 As shown in the figure, when a write request for a data block arrives, the server's all-flash storage array first calculates the hash fingerprint of the data block using a selected hash algorithm through its in-memory computing engine (such as SIMD acceleration). Using the generated fingerprint as the key, it queries the hash table in the memory layer to see if the hash fingerprint exists. If a target fingerprint identical to the fingerprint of the data block is found, the deduplication process is executed, updating the logical address to point to the physical address of the original data block instead of writing it again, and returning a successful write response. If a target fingerprint identical to the fingerprint of the data block is not found, the data block is written to the memory layer.

[0044] This embodiment limits online deduplication operations to the memory layer, where it enables online deduplication based on in-memory hash comparisons. Leveraging the fast retrieval capabilities of hash tables, this approach avoids large-scale reads from the hard disk, significantly reducing disk read overhead. Furthermore, by updating pointers rather than re-storing data blocks, this effectively reduces inefficient storage space usage and improves the space utilization and overall efficiency of the all-flash storage array.

[0045] In an exemplary embodiment, the data deduplication method further includes:

[0046] If the target fingerprint identical to the fingerprint of the data block to be written does not exist in the hash table, the data block to be written is written into the memory layer, the physical address of the data block to be written is obtained, a mapping relationship between the fingerprint of the data block to be written and the physical address is established, and the mapping relationship is inserted into the hash table.

[0047] Alternatively, as Figure 4As shown, if the target fingerprint matching the fingerprint of the data block to be written is not found, the data block to be written is written to the physical storage space of the flash array via a high-speed channel to obtain the physical address. A mapping relationship is then established between the fingerprint of the data block to be written and the physical address of the data block. This mapping relationship is then inserted into the hash table at the memory layer, and a write success signal is returned. The entire process utilizes a lock-free design, ensuring concurrency safety through atomic operations, ensuring stable performance even in high-throughput scenarios.

[0048] In this embodiment, when the fingerprint of a data block to be written fails to find a match in the hash table, the data block is written to the memory layer, and the physical address of the data block is simultaneously obtained and recorded. Subsequently, a mapping relationship between the fingerprint and the physical address is established and added to the hash table for subsequent deduplication queries. Through the on-the-fly mapping mechanism, the physical address of the new data block can be quickly assigned and the mapping established in the hash table, ensuring smooth write operations and avoiding write delays caused by data deduplication.

[0049] In an exemplary embodiment, the data deduplication method further includes:

[0050] According to the data blocks stored in the memory layer, the capacity of the hash table is dynamically adjusted based on the initial capacity of the hash table; the initial capacity refers to the capacity of the minimum data set generated when the storage system is started; when the capacity of the hash table is greater than a first capacity threshold, an asynchronous thread or process is started to increase the capacity of the hash table; the first capacity threshold is greater than the initial capacity; when the capacity of the hash table is greater than a second capacity threshold, the data blocks and corresponding fingerprints stored in the memory layer are asynchronously persisted to the first hard disk layer in the tree storage architecture; the second capacity threshold is greater than the first capacity threshold.

[0051] The hash table in this embodiment uses a dynamic expansion design. When the storage system starts, the hash table's capacity is the initial capacity. This initial capacity refers to the minimum storage capacity preset for the hash table at storage system startup. This capacity is set based on the minimum dataset size expected during system startup, aiming to provide a baseline capacity that balances execution efficiency and memory usage. For example, the hash table's initial capacity can be set to 5%-10% of system memory, and supports chained collision resolution or open addressing.

[0052] The first capacity threshold, also known as the soft threshold, triggers asynchronous hash table expansion. When hash table usage exceeds the soft threshold, the system initiates an asynchronous thread or process to preemptively increase hash table capacity to accommodate further data growth and avoid performance bottlenecks caused by insufficient capacity.

[0053] The second capacity threshold, also known as the hard threshold, is a higher storage capacity threshold than the soft threshold. When this threshold is reached, the system will perform a more stringent persistence operation, asynchronously flushing the data blocks and their fingerprint information stored in the memory tier to the first hard disk tier in the tree-like storage architecture to free up memory space and ensure stable system operation. For example, the first capacity threshold of the hash table is 80% occupancy, and the second capacity threshold is 95% occupancy. When the first capacity threshold is reached, asynchronous expansion is initiated, and when the second capacity threshold is reached, a synchronous flush (write) operation is triggered.

[0054] It's important to note that setting both soft and hard thresholds for the hash table, rather than just a hard threshold, is important for the following reasons: First, soft thresholds allow the system to begin preparing for capacity expansion when the hash table approaches its maximum capacity, rather than waiting until it's absolutely full. Once the hash table's fill rate reaches the soft threshold (e.g., 80%), the system can initiate asynchronous capacity expansion. This avoids sudden blockages or delays caused by hitting the hard threshold, ensuring smoother and more uninterrupted hash table operations for users. Second, capacity expansion is resource-intensive, involving steps like recalculating hash values ​​and redistributing data, which can cause performance degradation. If only a hard threshold is set, performance will fluctuate significantly whenever the hash table reaches full capacity. Using soft thresholds, however, allows the system to complete most capacity expansion work before the hard threshold is reached, mitigating the performance impact of the hard threshold being triggered.

[0055] Under certain conditions, such as before the system restarts or when the MemTable in the memory layer reaches a threshold, the changes in the hash table in memory are persisted to the hard disk to support system recovery and continuous deduplication operations. For example, as write operations continue, the MemTable in the memory layer will gradually reach the preset capacity threshold. At this time, the system will start the MemTable switching process: the currently active MemTable is marked as frozen, and all new write operations are redirected to the newly created MemTable instance. The frozen MemTable will be asynchronously persisted to the first hard disk layer in the tree storage architecture through a background thread and converted to SSTable format storage. During the persistence process, the system will sort and compress the fingerprint data to optimize the efficiency of subsequent queries.

[0056] To ensure data consistency, all MemTable entries are backed up in power-saving memory. This allows complete fingerprint index information to be restored even in the event of a system crash. Notably, the entire switchover process does not affect foreground write performance, achieving a smooth transition from memory to disk.

[0057] Through this embodiment, a reasonable initial capacity is set for the hash table to ensure efficient online deduplication services are available upon startup. Subsequently, the hash table capacity is dynamically adjusted based on the increase in data blocks stored in the memory layer, with two thresholds set: a first capacity threshold (soft threshold) and a second capacity threshold (hard threshold). When the hash table capacity exceeds the soft threshold, an asynchronous thread or process is launched to smoothly increase the hash table size to accommodate more data block fingerprint information. When the hard threshold is reached, the system asynchronously flushes the data blocks and their fingerprint information stored in the memory layer to the first hard disk layer (the LSM tree's L1 layer) and simultaneously clears the records of this data in the hash table. By dynamically adjusting the hash table capacity, the system can promptly expand storage space as data volume grows, avoiding performance degradation and data loss risks caused by hash table overflows. Furthermore, the asynchronous persistence strategy ensures data consistency between the memory layer and the hard disk layer. Even in the event of an abnormal system shutdown, the memory layer state can be restored using the data persisted on the hard disk, maintaining system stability and reliability.

[0058] In an exemplary embodiment, a hard disk layer in a plurality of hard disk layers stores a plurality of ordered data files; an ordered data file in the plurality of ordered data files maintains an independent filter and a fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of a plurality of data blocks in an ordered data file.

[0059] Ordered data files, or SSTable files, are the fundamental unit of data storage at the disk level in a tree-like storage architecture. Ordered data files store data sequentially by key value, supporting efficient data writing and batch reading. Each SSTable file contains data blocks within a specific range and their corresponding metadata, such as fingerprint information.

[0060] The filter is used for preliminary deduplication screening between SSTable files in the hard disk layer. It can quickly determine whether a data block may exist in a certain SSTable file, thereby reducing unnecessary hard disk read operations, improving the efficiency of background deduplication and reducing the I / O burden. For example, the filter can be a Bloom filter. If the fingerprint FX fails to pass the Bloom filter of a certain file (that is, it is marked as not existing), then the system can conclude that the fingerprint FX is not in this file, and there is no need to perform further precise fingerprint library comparison on the file, thereby saving I / O resources; if the fingerprint FX passes the Bloom filter of a certain file (that is, it is marked as existing), then the system can conclude that there is a possibility that the fingerprint FX matches the fingerprint FX in the file. At this time, the system needs to further query the fingerprint library of the file and perform precise fingerprint matching to confirm whether there is a real duplicate data block.

[0061] The fingerprint library refers to the set of data block fingerprints stored in each SSTable file. It is persisted in a B+ tree structure, supports efficient range queries and precise comparisons, and improves deduplication efficiency and accuracy. The fingerprint library is used to accurately identify and eliminate duplicate data blocks during background merging and deduplication, thereby improving storage space utilization and overall storage performance. In this embodiment, when the MemTable in the memory layer (L0) reaches a preset capacity threshold, the system triggers a MemTable flush operation to persist the data entries of the L0 layer to the L1 layer SSTable file on the hard disk. During the flush process, the data entries are sorted in the order of the key values, and the fingerprint metadata of the data blocks are also written to the SSTable file. This means that each SSTable file contains not only the actual data blocks, but also the fingerprint values ​​of these data blocks, which are used for subsequent data deduplication and query.

[0062] In some embodiments, when the first hard disk layer in the tree-like storage architecture satisfies the first trigger policy, a background deduplication operation is performed on the first hard disk layer according to the second deduplication policy, including:

[0063] An ordered data file among multiple ordered data files in the first hard disk layer is used as the current ordered data file, and the following deduplication operation is performed to perform background deduplication operation on the first hard disk layer: when the first hard disk layer in the tree-like storage architecture meets the first trigger strategy, the fingerprint library of the current ordered data file is compared with the fingerprint library in the adjacent ordered data file of the current ordered data file through the filter of the current ordered data file; when there is a target data block in the current ordered data file that has the same fingerprint as the data block in the adjacent ordered data file, the target data block is removed, and the pointer pointing to the physical address of the target data block is updated to obtain the deduplicated current ordered data file.

[0064] In some embodiments, after deduplication is performed on multiple ordered data files in the first hard disk layer, merging is performed on the multiple ordered data files in the first hard disk layer to obtain new ordered data files, and the new ordered data files are stored in the second hard disk layer in the tree-like storage architecture.

[0065] Among them, the current ordered data file refers to the SSTable file selected in the background merge and deduplication process. The target data block refers to the data block located in the current ordered data file (SSTable), whose fingerprint (hash value) matches the fingerprint of a data block in the adjacent ordered data file. This means that two or more data blocks have the same content and are therefore regarded as duplicate data. When the system recognizes the duplication of the target data block, it performs a deduplication operation, removes the data block in the current file, and updates all logical pointers pointing to its physical address to point to the only retained data block, thereby reducing storage space usage and optimizing storage efficiency. The process of updating all logical pointers pointing to its physical address is completed through atomic swaps to ensure that concurrent reads always see a consistent view of the data.

[0066] Adjacent ordered data files refer to a set of ordered data files (SSTables) in the tree-like storage architecture that are adjacent to the current ordered data file and sorted by key value. During background deduplication, the system compares the fingerprint of the current ordered data file with the fingerprint library of these adjacent files to achieve local deduplication and reduce unnecessary I / O operations.

[0067] For example, assume that the current ordered data file A and its adjacent ordered data file B both belong to the L1 layer of the tree-like storage architecture. During the background deduplication and merging process, the system first reads the Bloom filters in the current ordered data file A and the adjacent ordered data file B to perform a preliminary probabilistic screening of duplicate data blocks. For each data block in the current ordered data file A, the system checks whether its fingerprint may be included in the Bloom filter of the adjacent ordered data file B. If it passes the preliminary screening of the Bloom filter (i.e., it is marked as existing), it will further retrieve an exact match from the fingerprint library of the adjacent ordered data file B. For example, there is a data block X in the current ordered data file A, and its fingerprint is FX. If FX passes the Bloom filter detection of the adjacent ordered data file B, the system will search the fingerprint library of the adjacent ordered data file B to see if there is a fingerprint identical to FX. If it does not exist, data block X is considered unique. If it does exist, data block X and the data block with the same fingerprint in the adjacent ordered data file B are considered duplicates. The system deletes the duplicate data block X in the current ordered data file A and updates all logical address mappings pointing to the physical address of data block X to point to the unique data block with the same fingerprint in the adjacent ordered data file B. Finally, the system reorganizes and merges the contents of the current ordered data file A and the adjacent ordered data file B to form a new ordered data file C, which no longer contains any duplicate data blocks. Ordered data file C will replace the current ordered data file A and the adjacent ordered data file B and continue to participate in the subsequent merging and deduplication process.

[0068] In an embodiment of the present application, to reduce read and write overhead and improve efficiency, the system may select several adjacent files closest to the current file for fingerprint comparison, rather than limiting it to a single directly adjacent file. For example, assume that the current ordered data file A and four adjacent ordered data files B, C, D, and E all belong to the same L1 layer, and the system decides to perform a merge and deduplication operation involving multiple files. First, the system reads the Bloom filters of all files (A, B, C, D, and E) and loads them into memory, performing parallel membership checks to quickly eliminate impossible duplicate data blocks. For data block X in the current ordered data file A, whose fingerprint is FX, the system sequentially uses FX to test it against the Bloom filters of adjacent ordered data files B through E. If FX is marked as possibly present in any filter, the system further searches the fingerprint library of the corresponding file for an exact match. For example, assume that FX passes the initial test in the Bloom filters of adjacent ordered data file B and adjacent ordered data file D. Next, the system searches the fingerprint libraries of adjacent ordered data file B and adjacent ordered data file D for a fingerprint that matches FX. If a matching fingerprint FY (data block Y and data block X have the same content) is found in the adjacent ordered data file B but not in the adjacent ordered data file D, data block X is considered a duplicate of data block Y in the adjacent ordered data file B and is deleted. All logical address mappings pointing to data block X are updated to point to the retained data block Y in the adjacent ordered data file B. After the merge process is complete, the data in ordered data files A, B, C, D, and E are organized into new ordered data files G and H (multiple files may be generated depending on the merge strategy). F and G will contain no duplicate data blocks, ensuring optimal use of storage space. Ordered data files G and H will replace ordered data files A, B, C, D, and E and continue to participate in subsequent merge and deduplication processes.

[0069] Through this embodiment, a local deduplication strategy is adopted in the first hard disk layer. By limiting the comparison range (usually 2-4 files), a lightweight fingerprint library for comparing adjacent ordered data files is used instead of complete data blocks. Duplicate items are quickly identified through fingerprint comparison in memory, and I / O overhead is controlled within 1-2 times of the data required for a single merge, thereby reducing I / O overhead. Actual measurements show that its additional read amplification is only 1.3. In the local deduplication strategy, Bloom filters and fingerprint libraries are deeply combined for precise deduplication, thereby improving deduplication accuracy. After confirming duplicate data, the logical address is updated and redundant space is reclaimed, thereby improving storage space utilization and reducing the cost of large-scale data storage.

[0070] In an exemplary embodiment, when other hard disk layers except the first hard disk layer in the multiple hard disk layers meet the second trigger policy, background deduplication operations are performed on the other hard disk layers according to the third deduplication policy, including:

[0071] When other hard disk layers except the first hard disk layer in multiple hard disk layers meet the second trigger strategy, the hard disk status data of the other hard disk layers are input into a pre-trained cost model to obtain the predicted repetition rate of the other hard disk layers; when the predicted repetition rate is less than the repetition rate threshold, the fingerprint library of the multiple ordered data files stored in the other hard disk layers is compared to determine that there is a group of first data blocks with duplicate fingerprints in the multiple ordered data files stored in the other hard disk layers, delete the group of first data blocks, and update the pointer to the physical address of the group of first data blocks to obtain the multiple ordered data files in the other hard disk layers after deduplication.

[0072] In an exemplary embodiment, when the predicted repetition rate is greater than or equal to a repetition rate threshold, the fingerprint library of multiple ordered data files stored in other hard disk layers and the next hard disk layer of other hard disk layers is compared to determine that a group of fourth data blocks with duplicate fingerprints exist in the multiple ordered data files stored in the other hard disk layers, delete the group of fourth data blocks, and update the pointer to the physical address of the group of fourth data blocks to obtain multiple deduplicated ordered data files in the other hard disk layers.

[0073] In an exemplary embodiment, multiple deduplicated ordered data files in other hard disk layers are merged to obtain a merged ordered data file, and the merged ordered data file is stored in the next hard disk layer of the other hard disk layers.

[0074] Disk status data refers to the current status of all disk layers in the tree-like storage architecture, except for the memory layer (L0). For example, disk status data includes, but is not limited to, the number of SSTables at each layer, their total size, the percentage of free space, the write timestamps of data blocks, and historical data on the Bloom filter's false positive rate. Disk status data is used to assess the health and potential duplication rates of storage at different layers, providing a basis for triggering deduplication and merging operations.

[0075] The cost model is an algorithmic model used to predict and optimize the cost of deduplication operations at each disk layer in a tree-like storage architecture. Based on disk status data input and trained through machine learning or statistical analysis, it predicts expected I / O overhead, CPU utilization, and other associated costs under different deduplication strategies. It then assists decision-makers in selecting the optimal deduplication timing and tier based on the predicted duplication rate and cost-benefit ratio.

[0076] The predicted duplication rate, derived from analyzing drive status data using a cost model, indicates the expected proportion of data blocks that will be duplicated in an upcoming deduplication operation. This predicted duplication rate helps assess the cost-effectiveness of the current level of deduplication, guiding whether to trigger deduplication immediately or wait for a higher duplication rate to reduce I / O overhead.

[0077] The duplication rate threshold is a standard value set by the system administrator or automated policy. It is compared with the predicted duplication rate to determine whether to initiate deduplication and merging operations on the current disk tier. If the predicted duplication rate is less than the duplication rate threshold, deduplication and merging are initiated on the current disk tier. If the predicted duplication rate is greater than or equal to the duplication rate threshold, deduplication and merging are initiated across tiers.

[0078] A set of first data blocks is a series of data blocks that are confirmed to be duplicates through precise fingerprint matching during deduplication operations. A set of first data blocks has the same fingerprint across different ordered data files, indicating that they contain the same data content. During the background deduplication process, once duplication is confirmed, the system marks and removes these blocks (except for one instance) and updates all logical address mappings pointing to these blocks to reduce storage space usage and optimize storage efficiency.

[0079] A group of fourth data blocks specifically refers to a series of data blocks that have the same fingerprint and are therefore considered to be duplicates, identified through cross-layer fingerprint library comparison (i.e., comparison between the L2+ layer and other deeper layers, such as the SSTable files of the L3+ layer) when the predicted repetition rate is greater than or equal to the repetition rate threshold.

[0080] Optionally, when the system detects that a disk layer (L2+) other than the L1 layer (the first disk layer) meets the second trigger policy (for example, if the number of files reaches a threshold or storage space usage exceeds a certain limit), it begins collecting disk status data for that layer, including the number of files, total size, and percentage of free space. The system then feeds this collected L2+ disk status data into a pre-trained cost model, using data analysis to predict the duplication rate for the current layer and assess the cost-effectiveness of deduplication. If the predicted duplication rate is lower than a preset duplication rate threshold, the system performs deduplication within the layer, comparing only the fingerprints of the L2+ SSTable files. Through this comparison, the system identifies and marks a set of first data blocks with duplicate fingerprints, performs a deletion operation, and updates the logical address mapping pointer to the unique physical block address, thereby generating a deduplicated set of SSTable files. If the predicted duplication rate exceeds or equals the duplication rate threshold, indicating the possibility of highly duplicated data across a larger area, the system initiates a cross-layer deduplication process, comparing the fingerprints of the L2+ SSTables with the fingerprints of the SSTables in the next layer below the L2+ layer (L3+). This process also uses Bloom filter pre-screening to reduce invalid hard disk reads, and then performs precise fingerprint matching to find and mark a set of fourth data blocks with duplicate fingerprints for deduplication. For data blocks marked as duplicates (whether within a layer or across layers), the system will perform physical deletion and update all logical addresses pointing to these data blocks to point to the unique physical block location retained to ensure data consistency and efficient use of storage space. After the deduplication operation is completed, the system will perform a merge process to integrate the deduplicated SSTable files into a merged ordered data file, which is optimized and compressed. Finally, this optimized file will be stored in the next hard disk layer (such as the L3 layer) to complete the vertical migration of data and further optimize the storage layout and performance.

[0081] Through this embodiment, a cost model is introduced to predict the duplication rate. When the predicted duplication rate is lower than the threshold, deduplication is only performed in the current layer, avoiding unnecessary cross-layer comparisons and reducing I / O overhead. When the predicted duplication rate is higher than or equal to the threshold, the comparison is expanded downward to deeper layers, achieving accurate deduplication from a global perspective, greatly improving space utilization and data management efficiency, ensuring more thorough deduplication in high-duplication data scenarios, and thus greatly improving storage efficiency.

[0082] In an exemplary embodiment, a hard disk layer in a plurality of hard disk layers stores a plurality of ordered data files; an ordered data file in the plurality of ordered data files maintains an independent filter and a fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of a plurality of data blocks in an ordered data file.

[0083] The filter and the fingerprint library have been explained in the above embodiments and will not be repeated here.

[0084] In some embodiments, comparing the fingerprint library of multiple ordered data files stored in other hard disk layers includes:

[0085] Load filters of multiple ordered data files stored in other hard disk layers in parallel, and control batch judgment of filters of multiple ordered data files stored in other hard disk layers; when there is a group of second data blocks in multiple ordered data files stored in other hard disk layers, compare the fingerprints of data blocks other than the group of second data blocks in the fingerprint library of multiple ordered data files stored in other hard disk layers; a group of second data blocks refers to data blocks in multiple ordered data files stored in other hard disk layers that are not hit by the filters.

[0086] Among them, "not hit by the filter" refers to the situation where the fingerprint of the data block is not directly identified as possibly existing in the SSTable after being detected by the Bloom filter. A group of second data blocks specifically refers to a set of data blocks that are identified as not hit by the filter after being quickly screened by the Bloom filter during the background merge process. In actual tests, 60%-80% of non-duplicate blocks can be filtered out by the Bloom filter, reducing unnecessary hard disk read operations. A group of second data blocks are considered to be potential unique data blocks, that is, they may not be repeated in the SSTable level involved in the current merge operation. Due to the characteristics of the Bloom filter, the data blocks in a group of second data blocks can be directly determined as unique blocks without further fingerprint library comparison, thereby avoiding unnecessary hard disk read operations and significantly improving the efficiency of deduplication operations. Therefore, the present application utilizes the characteristics of the Bloom filter to filter non-duplicate blocks, and then performs a more detailed fingerprint library comparison in the data blocks hit by the filter, which can significantly improve the efficiency of the deduplication operation.

[0087] Optionally, when the background merge operation is started, the system uses multi-threading technology to load the Bloom filters of the SSTable files to be merged into the memory in parallel; using the SIMD instruction set or similar parallel computing technology, the system performs batch member detection on the fingerprint library of multiple ordered data files stored in other hard disk layers, and quickly determines which data blocks are not hit by the corresponding Bloom filters, that is, which blocks may be unique or less repeated. The data blocks that are not hit (that is, the filter return may not include) are further extracted to form a group of second data blocks. The system queries the global fingerprint library of the hard disk layer (such as the B+ tree structure), making full use of the efficient query mechanism of the B+ tree, comparing the fingerprints of other data blocks except the group of second data blocks, confirming the existence of duplicate data, reducing invalid comparisons, and ensuring the accuracy of deduplication.

[0088] Through this embodiment, the parallel loading and batch judgment capabilities of the Bloom filter are utilized to pre-screen possible unique data blocks to obtain a group of second data blocks. This group of second data blocks does not require further fingerprint library comparison, avoiding unnecessary hard disk read operations, and significantly improving the response speed of online deduplication and the throughput of the all-flash storage array; a dedicated Bloom filter is embedded in the header of the ordered data file (SSTable file) at the L1+ layer, which can quickly exclude non-duplicate blocks through the filter during the background merging process, and then perform fingerprint library comparison on the remaining possible duplicate data blocks. It combines the efficient pre-screening of the Bloom filter with the precise comparison of the B+ tree structure, effectively solving the high I / O overhead problem of traditional deduplication methods and improving the overall deduplication accuracy and efficiency.

[0089] In an exemplary embodiment, multiple ordered data files stored in other hard disk layers have corresponding historical false positive rates, where the historical false positive rates refer to the historical frequency at which the corresponding ordered data files are falsely judged by the corresponding filters to contain duplicate data blocks. Comparing the fingerprint libraries of the multiple ordered data files stored in other hard disk layers includes:

[0090] Determine a target ordered data file in other hard disk layers whose historical misjudgment rate is greater than a preset threshold, load filters of ordered data files other than the target ordered data file in other hard disk layers in parallel, and control batch judgment of filters of ordered data files other than the target ordered data file in other hard disk layers; when a group of third data blocks exists in ordered data files other than the target ordered data file in other hard disk layers, compare fingerprints of data blocks other than the group of third data blocks in a fingerprint library of multiple ordered data files stored in other hard disk layers; a group of third data blocks refers to data blocks in ordered data files other than the target ordered data file in other hard disk layers that are not hit by filters.

[0091] In order to deal with false positives in the Bloom filter, in this embodiment, each ordered data file corresponds to a historical false positive rate. The historical false positive rate specifically refers to the frequency with which the data blocks of the ordered data file were incorrectly identified as potentially duplicated (i.e., false positives) by the corresponding Bloom filter in the previous merge operation. In this embodiment, the historical data of the false positive rate is recorded, and the filter usage strategy in subsequent merges is dynamically adjusted (such as skipping filtering and directly comparing SSTables with high false positive rates).

[0092] Target ordered data files specifically refer to those ordered data files whose historical false positive rates exceed a preset threshold among multiple SSTable files stored in other disk tiers. For these target ordered data files, during background merging and deduplication operations, the system does not rely on their Bloom filters for preliminary screening. Instead, it directly performs precise comparisons against the fingerprint library to ensure that data deduplication accuracy is not affected by the high false positive rate.

[0093] A set of third data blocks refers to a collection of data blocks that are stored in SSTable files other than the target ordered data file in other hard disk layers (L1+ layer) during the background merging and data deduplication process and are not hit by the corresponding Bloom filter. In other words, a set of third data blocks has passed the initial duplication screening and has been identified by the Bloom filter as a potentially unique data block, so there is no need to immediately perform an accurate comparison with the fingerprint library. The existence of a set of third data blocks means that in the subsequent accurate deduplication process, the system only needs to focus on those data blocks that have not been directly screened by the Bloom filter, as well as all data blocks from the target ordered data files with a high misjudgment rate, thereby significantly reducing unnecessary hard disk read operations and optimizing the efficiency of data deduplication.

[0094] Optionally, the server regularly analyzes the misjudgment records of the Bloom filters of each SSTable file in the L1+ layer, and marks the files whose historical misjudgment rate exceeds a preset threshold as target ordered data files. When the background merge is started, the Bloom filters of all SSTables except the target ordered data file are loaded into the memory in parallel. For the data blocks involved in the current merge operation, batch member detection is performed through the Bloom filters in the memory to filter out a group of third data blocks, that is, a set of data blocks that are preliminarily judged to be non-duplicate. For other data blocks that are not directly confirmed as unique, the fingerprint information in the fingerprint library is compared, including all data block fingerprints in the target ordered data file, as well as a group of data block fingerprints other than the third data block. After confirming the duplicate data, the logical address is updated so that all duplicate data point to a unique physical block, and redundant storage space is reclaimed at the same time.

[0095] Through this embodiment, the historical misjudgment rate of the Bloom filter of the ordered data files in the hard disk layer is continuously monitored, and the target ordered data files whose misjudgment rate exceeds the preset threshold can be accurately identified, thereby solving the problem that the deduplication efficiency and accuracy may be reduced due to the increase in the misjudgment rate of the Bloom filter; the identified target files are directly accurately compared with the fingerprint library when merged, avoiding the false positive problem that may be caused by pre-screening.

[0096] In some embodiments, it can be seen from the above embodiments that the first hardware layer (L1 layer) adopts lightweight deduplication, and only compares the fingerprints of adjacent SSTables to reduce I / O overhead; other hardware layers (L2+ layers) combine Bloom filters and global fingerprint libraries for precise deduplication. Among them, the global fingerprint library refers to a database that centrally stores and manages the unique identifiers (fingerprints) of data blocks in order to achieve effective deduplication of all stored data, which contains the fingerprint information of all data blocks in the all-flash storage array. The global fingerprint library can adopt a B+ tree structure. In an all-flash storage array that stores a large amount of data, the size of the fingerprint library directly affects the memory usage efficiency and query performance. Therefore, in this embodiment, the above-mentioned data deduplication method also includes the following steps:

[0097] 1. Continuously monitor the load of the all-flash storage array and the characteristics of the written data. Data characteristics include repetition rate and type characteristics. Type characteristics refer to the properties of the written data, such as file size distribution and data block similarity, to determine the optimization direction of the global fingerprint library.

[0098] Second, based on the monitored load and data characteristics, the global fingerprint library optimization strategy is dynamically determined through intelligent algorithms (such as rule-based decision trees and machine learning models). The optimization strategy determines whether the global fingerprint library should be compressed or expanded. For example, if a high proportion of duplicate data is detected and the system load is low, the global fingerprint library will be expanded to accommodate more fingerprint information and improve deduplication accuracy. Otherwise, compression may be triggered.

[0099] 3. When the optimization strategy is used to compress the global fingerprint library, similar fingerprint entries are clustered. For multiple similar fingerprints generated by similar data blocks, multiple similar fingerprints are analyzed and merged to reduce the size of the fingerprint library while maintaining the accuracy of deduplication.

[0100] 4. When the optimization strategy represents the expansion of the global fingerprint library, at the memory layer, the capacity of the hash table in the memory is dynamically increased as needed to store the fingerprint information of more data blocks; at the hard disk layer, additional index levels are introduced, such as using a tree-based index structure (such as B+ tree) to improve the query efficiency of large-scale fingerprint libraries.

[0101] 5. Continuously monitor the performance indicators of the fingerprint library after operation, such as query latency, disk I / O, memory usage, etc., and adjust the compression and expansion strategies of the fingerprint library based on the feedback loop of these indicators to ensure the optimization of storage efficiency and query performance.

[0102] Through this embodiment, by continuously monitoring the data repetition rate and type characteristics and dynamically adjusting the expansion or compression of the fingerprint library, storage redundancy can be effectively reduced and storage space utilization can be significantly improved. In particular, for data with high repetition rates, the precise expansion of the fingerprint library can ensure more comprehensive deduplication, avoid unnecessary storage occupation, and thus reduce storage costs. Under dynamic optimization based on system load and data characteristics, the fingerprint library at the hard disk layer improves query efficiency under large-scale data by introducing additional index levels (such as B+ trees), reduces disk I / O operations, shortens query latency, and improves the overall response speed of the system. At the same time, the intelligent expansion of the hash table at the memory layer ensures the rapid location and processing of frequently accessed data, further optimizing system performance.

[0103] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0104] The embodiment of the present application also provides a data deduplication device, such as Figure 5 Shown, including:

[0105] The first deduplication module 502 is used to perform online deduplication operations on the memory layer according to the first deduplication strategy before writing the data block to be written into the memory layer of the tree-like storage architecture; the tree-like storage architecture includes multiple storage layers; the memory layer refers to the first storage layer among the multiple storage layers, which is used to accelerate the writing of the data block to be written and support online deduplication; the multiple storage layers other than the first storage layer in the multiple storage layers are multiple hard disk layers, which are used for background global deduplication and long-term data storage; the memory layer and the multiple hard disk layers adopt different deduplication strategies.

[0106] The second deduplication module 504 is configured to perform background deduplication on the first hard disk layer in the tree-like storage architecture according to the second deduplication strategy when the first hard disk layer satisfies the first trigger strategy.

[0107] The third deduplication module 506 is configured to perform background deduplication operations on the other hard disk layers except the first hard disk layer in the multiple hard disk layers according to the third deduplication strategy when the other hard disk layers in the multiple hard disk layers meet the second trigger strategy.

[0108] In an exemplary embodiment, the memory layer stores a hash table; the hash table stores the fingerprint of the data block stored in the memory layer and a pointer to the physical address of the data block; the first deduplication module 502 is also used to calculate the fingerprint of the data block to be written before writing the data block to the memory layer, and query the hash table. If there is a target fingerprint in the hash table that is the same as the fingerprint of the data block to be written, the pointer in the hash table pointing to the physical address of the data block corresponding to the target fingerprint is updated.

[0109] In an exemplary embodiment, the first deduplication module 502 is also used to write the data block to be written into the memory layer when there is no target fingerprint identical to the fingerprint of the data block to be written in the hash table, obtain the physical address of the data block to be written, establish a mapping relationship between the fingerprint of the data block to be written and the physical address, and insert the mapping relationship into the hash table.

[0110] In an exemplary embodiment, the first deduplication module 502 is also used to dynamically adjust the capacity of the hash table based on the initial capacity of the hash table according to the data blocks stored in the memory layer; the initial capacity refers to the capacity of the minimum data set generated when the storage system is started; when the capacity of the hash table is greater than the first capacity threshold, an asynchronous thread or process is started to increase the capacity of the hash table; the first capacity threshold is greater than the initial capacity; when the capacity of the hash table is greater than the second capacity threshold, the data blocks and corresponding fingerprints stored in the memory layer are asynchronously persisted to the first hard disk layer in the tree-like storage architecture; the second capacity threshold is greater than the first capacity threshold.

[0111] In an exemplary embodiment, a hard disk layer in a plurality of hard disk layers stores a plurality of ordered data files; an ordered data file in the plurality of ordered data files maintains an independent filter and fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of a plurality of data blocks in an ordered data file; the second deduplication module 504 is also used to use an ordered data file in the plurality of ordered data files in the first hard disk layer as the current ordered data file, and perform the following deduplication operation to perform background deduplication operation on the first hard disk layer: when the first hard disk layer in the tree-like storage architecture satisfies the first trigger strategy, the fingerprint library of the current ordered data file is compared with the fingerprint library in the adjacent ordered data file of the current ordered data file through the filter of the current ordered data file; when there is a target data block in the current ordered data file that has the same fingerprint as the data block in the adjacent ordered data file, the target data block is removed, and the pointer pointing to the physical address of the target data block is updated to obtain the deduplicated current ordered data file.

[0112] In an exemplary embodiment, the second deduplication module 504 is also used to perform merging processing on the multiple ordered data files in the first hard disk layer after performing deduplication operations on the multiple ordered data files in the first hard disk layer to obtain new ordered data files, and store the new ordered data files in the second hard disk layer in the tree-like storage architecture.

[0113] In an exemplary embodiment, the third deduplication module 506 is also used to input the hard disk status data of other hard disk layers into a pre-trained cost model to obtain the predicted repetition rate of other hard disk layers when other hard disk layers except the first hard disk layer in multiple hard disk layers meet the second trigger strategy; when the predicted repetition rate is less than the repetition rate threshold, compare the fingerprint library of multiple ordered data files stored in other hard disk layers to determine whether a group of first data blocks with duplicate fingerprints exist in the multiple ordered data files stored in other hard disk layers, delete the group of first data blocks, and update the pointer to the physical address of the group of first data blocks to obtain the deduplicated multiple ordered data files in the other hard disk layers.

[0114] In an exemplary embodiment, a hard disk layer in a plurality of hard disk layers stores a plurality of ordered data files; an ordered data file in a plurality of ordered data files maintains an independent filter and fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of a plurality of data blocks in an ordered data file; the third deduplication module 506 is also used to load filters of a plurality of ordered data files stored in other hard disk layers in parallel, and control batch judgment of filters of a plurality of ordered data files stored in other hard disk layers; when there is a group of second data blocks in the plurality of ordered data files stored in other hard disk layers, the fingerprints of the data blocks other than the group of second data blocks in the fingerprint library of the plurality of ordered data files stored in the other hard disk layers are compared; a group of second data blocks refers to data blocks in the plurality of ordered data files stored in other hard disk layers that are not hit by the filter.

[0115] In an exemplary embodiment, a hard disk layer in a plurality of hard disk layers stores a plurality of ordered data files; the ordered data files in the plurality of ordered data files are maintained with independent filters and fingerprint libraries; the filters are used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of a plurality of data blocks in an ordered data file; the plurality of ordered data files stored in other hard disk layers have corresponding historical false positive rates, and the historical false positive rate refers to the historical frequency at which the corresponding ordered data files are falsely judged by the corresponding filters to contain duplicate data blocks; the third deduplication module 506 is further used to determine a target ordered data file in the other hard disk layers whose historical false positive rate is greater than a preset threshold, load filters for ordered data files other than the target ordered data file in the other hard disk layers in parallel, and control batch judgment of filters for ordered data files other than the target ordered data file in the other hard disk layers; if a group of third data blocks exists in ordered data files other than the target ordered data file in the other hard disk layers, compare the fingerprints of the data blocks other than the group of third data blocks in the fingerprint libraries of the plurality of ordered data files stored in the other hard disk layers; the group of third data blocks refers to data blocks in ordered data files other than the target ordered data file in the other hard disk layers that are not hit by the filters.

[0116] In an exemplary embodiment, the third deduplication module 506 is also used to compare the fingerprint libraries of multiple ordered data files stored in other hard disk layers and the next hard disk layer of other hard disk layers when the predicted repetition rate is greater than or equal to the repetition rate threshold, determine that a group of fourth data blocks with duplicate fingerprints exist in the multiple ordered data files stored in the other hard disk layers, delete the group of fourth data blocks, and update the pointer to the physical address of the group of fourth data blocks to obtain multiple ordered data files after deduplication in the other hard disk layers.

[0117] In an exemplary embodiment, the third deduplication module 506 is also used to merge multiple deduplicated ordered data files in other hard disk layers to obtain a merged ordered data file, and store the merged ordered data file in the next hard disk layer of other hard disk layers.

[0118] For the description of the features in the embodiment corresponding to the data deduplication device, please refer to the relevant description of the embodiment corresponding to the data deduplication method, and will not be repeated here.

[0119] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned data deduplication method embodiments.

[0120] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned data deduplication method embodiments when run.

[0121] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0122] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data deduplication method embodiments are implemented.

[0123] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps in any of the above-mentioned data deduplication method embodiments.

[0124] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0125] The above is a detailed introduction to a data deduplication method, device, storage medium, electronic device and program product provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data deduplication method, characterized in that: include: Before writing the data block to be written into the memory layer of the tree-like storage architecture, performing an online deduplication operation on the memory layer according to a first deduplication strategy; The tree-like storage architecture includes multiple storage layers; The memory layer refers to the first storage layer among the multiple storage layers, which is used to accelerate the writing of the data blocks to be written and support online deduplication; the multiple storage layers other than the first storage layer among the multiple storage layers are multiple hard disk layers, which are used for background global deduplication and long-term data storage; The memory layer and the multiple hard disk layers adopt different deduplication strategies; When the first hard disk layer in the tree-like storage architecture satisfies the first trigger policy, performing background deduplication operation on the first hard disk layer according to the second deduplication policy; When other hard disk layers except the first hard disk layer in the multiple hard disk layers meet the second trigger strategy, performing background deduplication operations on the other hard disk layers according to the third deduplication strategy; Among them, the memory layer stores a hash table; the hash table stores the fingerprint of the data block stored in the memory layer and the pointer to the physical address of the data block; before writing the data block to be written into the memory layer of the tree-like storage architecture, the memory layer is subjected to an online deduplication operation according to the first deduplication strategy, including: before writing the data block to be written into the memory layer, calculating the fingerprint of the data block to be written, and querying the hash table, and when there is a target fingerprint in the hash table that is the same as the fingerprint of the data block to be written, updating the pointer in the hash table pointing to the physical address of the data block corresponding to the target fingerprint.

2. The method according to claim 1, characterized in that The method further comprises: If there is no target fingerprint identical to the fingerprint of the data block to be written in the hash table, the data block to be written is written into the memory layer, the physical address of the data block to be written is obtained, a mapping relationship between the fingerprint of the data block to be written and the physical address is established, and the mapping relationship is inserted into the hash table.

3. The method according to claim 1 or 2, characterized in that The method further comprises: Dynamically adjust the capacity of the hash table based on the initial capacity of the hash table according to the data blocks stored in the memory layer; the initial capacity refers to the capacity of the minimum data set generated when the storage system is started; In a case where the capacity of the hash table is greater than a first capacity threshold, starting an asynchronous thread or process to increase the capacity of the hash table; the first capacity threshold is greater than the initial capacity; When the capacity of the hash table is greater than a second capacity threshold, the data blocks and corresponding fingerprints stored in the memory layer are asynchronously persisted to the first hard disk layer in the tree-like storage architecture; the second capacity threshold is greater than the first capacity threshold.

4. The method according to claim 1, wherein A hard disk layer in the plurality of hard disk layers stores a plurality of ordered data files; an ordered data file in the plurality of ordered data files maintains an independent filter and a fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of the plurality of data blocks in the ordered data file; When the first hard disk layer in the tree-like storage architecture satisfies the first trigger policy, performing a background deduplication operation on the first hard disk layer according to the second deduplication policy, including: An ordered data file among the multiple ordered data files in the first hard disk layer is used as a current ordered data file, and the following deduplication operation is performed to perform background deduplication operation on the first hard disk layer: When the first hard disk layer in the tree-like storage architecture satisfies the first triggering strategy, the fingerprint library of the current ordered data file is compared with the fingerprint libraries in the adjacent ordered data files of the current ordered data file through the filter of the current ordered data file; When there is a target data block in the current ordered data file that has the same fingerprint as a data block in an adjacent ordered data file, the target data block is removed, and a pointer pointing to the physical address of the target data block is updated to obtain the deduplicated current ordered data file.

5. The method according to claim 4, characterized in that The method further comprises: After performing deduplication operations on the multiple ordered data files in the first hard disk layer, performing merging processing on the multiple ordered data files in the first hard disk layer to obtain new ordered data files, and storing the new ordered data files in the second hard disk layer in the tree-like storage architecture.

6. The method according to claim 1, characterized in that When the other hard disk layers except the first hard disk layer in the multiple hard disk layers meet the second trigger strategy, performing background deduplication operations on the other hard disk layers according to the third deduplication strategy, including: When other hard disk layers except the first hard disk layer in the multiple hard disk layers meet the second trigger strategy, inputting hard disk status data of the other hard disk layers into a pre-trained cost model to obtain predicted repetition rates of the other hard disk layers; When the predicted repetition rate is less than the repetition rate threshold, the fingerprint library of the multiple ordered data files stored in the other hard disk layer is compared to determine that a group of first data blocks with duplicate fingerprints exist in the multiple ordered data files stored in the other hard disk layer, the group of first data blocks is deleted, and the pointer pointing to the physical address of the group of first data blocks is updated to obtain the multiple ordered data files in the other hard disk layer after deduplication.

7. The method according to claim 6, characterized in that A hard disk layer in the plurality of hard disk layers stores a plurality of ordered data files; an ordered data file in the plurality of ordered data files maintains an independent filter and a fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of the plurality of data blocks in the ordered data file; The comparing the fingerprint library of the plurality of ordered data files stored in the other hard disk layer includes: Loading filters of the plurality of ordered data files stored in the other hard disk layers in parallel, and controlling batch judgment of filters of the plurality of ordered data files stored in the other hard disk layers; In the case where there is a group of second data blocks in the multiple ordered data files stored in the other hard disk layer, the fingerprints of the data blocks other than the group of second data blocks in the fingerprint library of the multiple ordered data files stored in the other hard disk layer are compared; the group of second data blocks refers to the data blocks in the multiple ordered data files stored in the other hard disk layer that are not hit by the filter.

8. The method according to claim 6, characterized in that A hard disk layer in the multiple hard disk layers stores multiple ordered data files; an ordered data file in the multiple ordered data files maintains an independent filter and fingerprint library; the filter is used to determine whether a data block in a corresponding ordered data file is in another ordered data file; the fingerprint library refers to a collection of fingerprints of multiple data blocks in the ordered data file; the multiple ordered data files stored in the other hard disk layers have corresponding historical false positive rates, and the historical false positive rates refer to the historical frequency at which the corresponding ordered data files are falsely judged by the corresponding filters to contain duplicate data blocks; the fingerprint library for comparing the multiple ordered data files stored in the other hard disk layers includes: Determine a target ordered data file in the other hard disk layers whose historical misjudgment rate is greater than a preset threshold, load filters of ordered data files other than the target ordered data file in the other hard disk layers in parallel, and control batch judgment of filters of ordered data files other than the target ordered data file in the other hard disk layers; In the case that there is a group of third data blocks in the ordered data files other than the target ordered data file in the other hard disk layers, the fingerprints of the data blocks other than the group of third data blocks in the fingerprint library of multiple ordered data files stored in the other hard disk layers are compared; the group of third data blocks refers to the data blocks in the ordered data files other than the target ordered data file in the other hard disk layers that are not hit by the filter.

9. The method according to claim 6, characterized in that The method further comprises: When the predicted repetition rate is greater than or equal to the repetition rate threshold, the fingerprint library of the multiple ordered data files stored in the other hard disk layer and the next hard disk layer of the other hard disk layer is compared to determine that a group of fourth data blocks with duplicate fingerprints exist in the multiple ordered data files stored in the other hard disk layer, delete the group of fourth data blocks, and update the pointer to the physical address of the group of fourth data blocks to obtain the multiple ordered data files after deduplication in the other hard disk layer.

10. The method according to claim 6, characterized in that The method further comprises: The multiple ordered data files after deduplication in the other hard disk layers are merged to obtain a merged ordered data file, and the merged ordered data file is stored in the next hard disk layer of the other hard disk layers.

11. A data deduplication device, characterized in that: include: A first deduplication module is configured to perform an online deduplication operation on the memory layer according to a first deduplication strategy before writing the data block to be written into the memory layer of the tree-like storage architecture; The tree-like storage architecture includes multiple storage layers; The memory layer refers to the first storage layer among the multiple storage layers, which is used to accelerate the writing of the data blocks to be written and support online deduplication; the multiple storage layers other than the first storage layer among the multiple storage layers are multiple hard disk layers, which are used for background global deduplication and long-term data storage; the memory layer and the multiple hard disk layers adopt different deduplication strategies; Before writing a data block to be written into the memory layer, calculating the fingerprint of the data block to be written, querying a hash table, and if a target fingerprint identical to the fingerprint of the data block to be written exists in the hash table, updating a pointer in the hash table pointing to the physical address of the data block corresponding to the target fingerprint; The memory layer stores the hash table; the hash table stores the fingerprint of the data block stored in the memory layer and the pointer to the physical address of the data block; a second deduplication module, configured to perform background deduplication on the first hard disk layer in the tree-like storage architecture according to the second deduplication strategy when the first hard disk layer satisfies the first trigger strategy; The third deduplication module is used to perform background deduplication operations on the other hard disk layers except the first hard disk layer in the multiple hard disk layers according to the third deduplication strategy when the other hard disk layers meet the second trigger strategy.

12. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the data deduplication method according to any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data deduplication method according to any one of claims 1 to 10.

14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data deduplication method according to any one of claims 1 to 10 are implemented.

Images