A data deduplication method, device, equipment and medium

Abstract

The application discloses a data deduplication method and device, equipment and medium, and relates to the technical field of storage. When an I / O request issued by a host is received, the optimal deduplication strategy can be adaptively selected in combination with the corresponding hardware resource utilization threshold according to the hardware resource utilization under the current I / O load, and data deduplication is performed, so that the dynamic adaptation to the I / O load change is realized, the deduplication rate is guaranteed without sacrificing the storage performance, the storage space is saved, the storage performance is optimized, and the storage system has higher elastic expansion capability.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a data deduplication method, apparatus, device, and medium. Background Technology

[0002] Data deduplication is a core technology that optimizes storage space by identifying and eliminating redundant data. Its basic principle is to replace duplicate copies with unique data blocks, thereby reducing storage resource redundancy. This technology is widely used in backup systems, cloud storage, virtualization, and cold data archiving. Deduplication efficiency is closely related to system resources such as the Central Processing Unit (CPU), memory, storage input / output (I / O), and network bandwidth.

[0003] However, current deduplication technologies typically rely on static parameter configuration or simple practical adjustments, making it difficult to dynamically adapt to changes in I / O load and failing to adequately consider the performance bottlenecks of the storage system. This leads to storage performance degradation under high load due to the computational overhead of deduplication, while under low load, hardware resource utilization is insufficient, and there is a lack of elastic scalability.

[0004] Given the above, how to solve the problem that current deduplication solutions cannot dynamically adapt to changes in I / O load, resulting in decreased storage performance under high load, insufficient resource utilization under low load, and lack of elastic scalability, is an urgent issue for technical personnel in this field. Summary of the Invention

[0005] This invention provides a data deduplication method, apparatus, device, and medium to at least solve the problem that current deduplication schemes cannot dynamically adapt to changes in I / O load, resulting in decreased storage performance under high load, insufficient resource utilization under low load, and a lack of elastic scalability.

[0006] This invention provides a data deduplication method applied to a storage system; the method includes:

[0007] When an input / output request is received from the host side, the hardware resource utilization is monitored according to the first preset period; wherein, the hardware resource utilization includes at least the central processing unit utilization, memory utilization, inter-node interconnect bandwidth utilization, and back-end disk utilization.

[0008] Obtain the hardware resource utilization threshold corresponding to the utilization rate of each hardware resource;

[0009] The deduplication strategy is determined based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, and data deduplication is performed according to the deduplication strategy and input / output requests.

[0010] The present invention also provides a data deduplication device for use in a storage system; the device includes:

[0011] The monitoring module is used to monitor hardware resource utilization according to a first preset period when it receives an input / output request from the host side; wherein, hardware resource utilization includes at least the central processing unit utilization, memory utilization, inter-node interconnect bandwidth utilization, and back-end disk utilization.

[0012] The acquisition module is used to acquire the hardware resource utilization threshold corresponding to the utilization rate of each hardware resource.

[0013] The deduplication module is used to determine the deduplication strategy based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, and to perform data deduplication according to the deduplication strategy and input / output requests.

[0014] The present invention also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the steps of any of the above-described data deduplication methods when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data deduplication methods.

[0016] The beneficial effects of this invention are that when an I / O request is received from the host side, it can adaptively select the optimal deduplication strategy and perform data deduplication based on the hardware resource utilization rate under the current I / O load and the corresponding hardware resource utilization threshold. This achieves dynamic adaptation to changes in I / O load, ensures the deduplication rate without sacrificing storage performance, saves storage space, optimizes storage performance, and enables the storage system to have higher elastic scalability.

[0017] In addition, the present invention also provides a data deduplication device, equipment and medium, with the same effect as above. Attached Figure Description

[0018] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of a data deduplication architecture provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart of a data deduplication method provided in an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of a data deduplication device provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0023] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Current deduplication technologies typically rely on static parameter configuration or simple practical adjustments, making it difficult to dynamically adapt to changes in I / O load and failing to adequately consider the performance bottlenecks of the storage system. This leads to storage performance degradation under high load due to the computational overhead of deduplication, while under low load, hardware resource utilization is insufficient, and there is a lack of elastic scalability. Therefore, to address these issues, this invention provides a data deduplication method.

[0026] It should be noted that the method provided by this invention is applied to a storage system. The host utilizes a switch to connect to the storage system via protocols such as Internet Small Computer System Interface (iSCSI), Fibre Channel - Small Computer System Interface (FC-SCSI), and Fibre Channel - Non-Volatile Memory Express (FC-NVME), forming a Storage Area Network (SAN) system, and issues I / O to the storage system. The storage system is responsible for processing and scheduling deduplication I / O, and storing deduplicated data.

[0027] Figure 1 This is a schematic diagram of a data deduplication architecture provided in an embodiment of the present invention. Figure 1 As shown, the data deduplication method in the storage system is jointly executed by four modules: a configuration module, a real-time resource monitoring module, a multi-dimensional deduplication feature selection module, and a deduplication execution module. The data deduplication method will be explained in detail below, referring to each module.

[0028] Figure 2 This is a flowchart illustrating a data deduplication method provided in an embodiment of the present invention. Figure 2 As shown, the method includes:

[0029] S10: When an input / output request is received from the host side, monitor the hardware resource utilization rate according to the first preset cycle.

[0030] Among them, hardware resource utilization includes at least the utilization of the central processing unit, memory utilization, inter-node interconnect bandwidth utilization, and back-end disk utilization.

[0031] Specifically, after the host sends I / O to the storage system, the real-time resource monitoring module monitors the hardware resource utilization according to a first preset period. It should be noted that this embodiment does not limit the first preset period; for example, it can be 1 second. Furthermore, hardware resource utilization includes at least CPU utilization, memory utilization, inter-node interconnect bandwidth utilization, and backend disk utilization, and may also include other types of hardware resource utilization, depending on the specific implementation.

[0032] It should be noted that CPU utilization can be obtained using the `top` command. The CPU utilization is the ratio of the percentage of CPU used by the main processing logic process (plmain) to the number of CPU cores. Memory utilization is the ratio of the memory used by the current plmain process to the total memory size. Inter-node bandwidth utilization is the ratio of the current node's interconnect bandwidth to the total node interconnect bandwidth. Backend disk utilization is the percentage of disk busy time; for example, in a striped RAID 5, the time taken for a single disk to process I / O within 1 second is the backend disk utilization.

[0033] S11: Obtain the hardware resource utilization threshold corresponding to the utilization rate of each hardware resource.

[0034] Furthermore, the real-time resource monitoring module obtains hardware resource utilization thresholds pre-configured by the configuration module, corresponding to the utilization rates of each hardware resource. For example, if hardware resource utilization includes at least CPU utilization, memory utilization, inter-node interconnect bandwidth utilization, and backend disk utilization, then the corresponding hardware resource utilization thresholds should at least include CPU utilization thresholds, memory utilization thresholds, inter-node interconnect bandwidth utilization thresholds, and backend disk utilization thresholds. In this embodiment, the specific size of each hardware resource utilization threshold is not limited and depends on the specific implementation.

[0035] S12: Determine the deduplication strategy based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, and perform data deduplication based on the deduplication strategy and input / output requests.

[0036] After acquiring the hardware resource utilization rate and corresponding threshold, the real-time resource monitoring module reports the hardware resource utilization rate and corresponding threshold to the multi-dimensional deduplication feature selection module when the hardware resource utilization rate exceeds the corresponding threshold. The multi-dimensional deduplication feature selection module has a built-in multi-dimensional deduplication feature selection function that can determine the deduplication strategy based on each hardware resource utilization rate and its corresponding threshold, and then report the deduplication strategy to the deduplication execution module. It should be noted that this embodiment does not restrict the specific process of determining the deduplication strategy based on each hardware resource utilization rate and its corresponding threshold.

[0037] Finally, the deduplication execution module incorporates a deduplication field, a hash algorithm, and a hash fingerprint index. After obtaining the deduplication strategy selected by the multi-dimensional deduplication feature selection module, it can perform data deduplication based on the strategy and I / O requests. In essence, the data deduplication process involves calculating the hash fingerprint value of the data block during I / O request processing to check if a data block with the same fingerprint already exists in the storage system. If the fingerprint already exists, there's no need to restore the data block; instead, a pointer is added to the logical block address (LBA) of the already stored data block, pointing to the new metadata, thus preventing duplicate data from being written to disk and improving disk utilization.

[0038] In this embodiment, when an I / O request is received from the host side, the system can adaptively select the optimal deduplication strategy and perform data deduplication based on the hardware resource utilization rate under the current I / O load and the corresponding hardware resource utilization threshold. This achieves dynamic adaptation to changes in I / O load, ensuring the deduplication rate without sacrificing storage performance, saving storage space, optimizing storage performance, and enabling the storage system to have higher elastic scalability.

[0039] To improve the performance of data deduplication, based on the above embodiments, in some embodiments, before monitoring the hardware resource utilization rate in the system according to the first preset period, the following is also included:

[0040] S13: Configure the initial hardware resource utilization threshold corresponding to the utilization rate of each hardware resource.

[0041] The initial hardware resource utilization thresholds include at least the initial CPU utilization threshold, the initial memory utilization threshold, the initial inter-node interconnect bandwidth utilization threshold, and the initial back-end disk utilization threshold.

[0042] S14: Determine the overall state and hardware type of the system.

[0043] The overall status includes at least single-controller clusters, multi-controller clusters, ordinary clusters, and dual-active clusters; the hardware type includes at least the CPU model and the back-end disk type.

[0044] S15: Adjust the initial hardware resource utilization thresholds according to the overall status and hardware type to generate the hardware resource utilization thresholds.

[0045] Specifically, when configuring hardware resource utilization thresholds in the configuration module, the initial hardware resource utilization thresholds corresponding to each hardware resource utilization are first configured. It can be understood that the initial hardware resource utilization thresholds include at least the initial CPU utilization threshold, the initial memory utilization threshold, the initial inter-node interconnect bandwidth utilization threshold, and the initial backend disk utilization threshold.

[0046] Furthermore, the overall system status and hardware type are determined. It should be noted that the overall status includes at least single-controller clusters, multi-controller clusters, ordinary clusters, and active-active clusters. A single-controller cluster is a cluster managed by a single controller; its structure is simple but it carries the risk of a single point of failure. A multi-controller cluster contains multiple controllers, improving system availability and performance through load balancing and failover. An ordinary cluster typically refers to a cluster with basic redundancy and fault recovery capabilities, and may include single-controller or multi-controller structures. An active-active cluster consists of two or more nodes that are simultaneously active, jointly processing requests to provide high availability and load balancing. Meanwhile, the hardware type includes at least the CPU model and backend disk type, and may also include other types.

[0047] It's worth noting that the configuration module has a built-in deduplication feature offset function. After obtaining the overall status of the storage system, it pre-determines the sensitive features of the storage and adjusts the initial hardware resource utilization thresholds based on the overall status and hardware type. Specifically, it offsets the initial hardware resource utilization thresholds by a certain percentage to generate the final hardware resource utilization thresholds, thereby effectively improving the performance of data deduplication. A detailed explanation follows:

[0048] In some embodiments, adjusting the initial hardware resource utilization thresholds based on the overall state and hardware type includes:

[0049] S151: When the overall state is a single-controller cluster, increase the initial inter-node interconnection bandwidth utilization threshold.

[0050] S152: When the overall state is a dual-active cluster, reduce the initial CPU utilization threshold.

[0051] S152: When the CPU model is the target model, increase the initial CPU utilization threshold.

[0052] S153: When the back-end disk type is mechanical hard disk, reduce the initial back-end disk utilization threshold.

[0053] Specifically, when the overall state of the storage system is determined to be a single-controller cluster, since there is no inter-node communication in a single-controller cluster, the inter-node communication bandwidth utilization threshold can be increased; while a dual-active cluster focuses more on real-time synchronization between data and has high requirements for I / O processing speed, so the CPU utilization threshold should be appropriately reduced to improve data deduplication efficiency.

[0054] Furthermore, the CPU model is obtained. If the CPU model is the target model, the initial CPU utilization threshold is increased. In this embodiment, there is no restriction on the target model. Finally, the type of the backend disk is obtained. When the backend disk type is a mechanical hard disk, fragmented data blocks will cause disk addressing difficulties, seriously affecting disk performance. In this case, the backend disk utilization threshold can be reduced to reduce the deduplication rate and avoid excessive data block fragmentation.

[0055] Based on the above embodiments, in some embodiments, a deduplication strategy is determined according to the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, including:

[0056] S120: Determine the deduplication timing, deduplication range, block partitioning strategy, hash algorithm, and hot / cold data index based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold.

[0057] Among them, the deduplication timing includes online deduplication and background deduplication; the deduplication scope includes global deduplication and local deduplication; the block division strategy includes fixed block division and variable block division; the hash algorithm includes a first hash algorithm and a second hash algorithm, with the collision rate and calculation speed of the first hash algorithm being lower than those of the second hash algorithm.

[0058] To determine the deduplication strategy, this embodiment specifically determines the deduplication timing, deduplication range, block partitioning strategy, hash algorithm, and hot / cold data index based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold.

[0059] It should be noted that deduplication timing includes online deduplication and background deduplication. Online deduplication is performed in real time when data is written to the storage system, reducing storage space usage but potentially increasing write latency. Background deduplication is performed by a background task after data is written, without affecting write performance but potentially causing temporary storage space expansion. Deduplication scope includes global deduplication and local deduplication. Global deduplication is performed throughout the entire storage system, identifying and eliminating all duplicate data blocks to minimize storage space usage. Local deduplication is performed within a specific scope (such as a single volume or directory), suitable for specific application scenarios and reducing computational overhead. Blocking strategies include fixed-size blocking and variable-size blocking. Fixed-size blocking divides data into fixed-size blocks for deduplication, which is simple to implement but may not effectively handle data changes. Variable-size blocking dynamically adjusts the block size based on the data content, improving deduplication efficiency, especially suitable for scenarios with frequent data changes. Hash algorithms include a first hash algorithm and a second hash algorithm. The collision rate and computation speed of the first hash algorithm are lower than those of the second hash algorithm, respectively. This embodiment does not limit the specific types of the first and second hash algorithms. For example, the first hash algorithm could be SHA-256, and the second hash algorithm could be xxHash64+xxHash3. The first hash algorithm offers high security and an extremely low collision probability, making it suitable for security-sensitive applications, but its computation speed is relatively slow. The second hash algorithm offers extremely fast hash calculation speed, making it suitable for scenarios with high performance requirements, but its collision rate is relatively high. Finally, cold and hot data indexing refers to the optimization of tiered indexing for cold and hot data. Specifically, tiered cold and hot data divides data into cold and hot data based on data access frequency, storing them on different storage media to optimize storage performance and cost. At the same time, by optimizing the index structure, the retrieval efficiency of hash fingerprints is improved, and the computational overhead of deduplication operations is reduced.

[0060] Therefore, this solution determines which deduplication strategy to use based on the sensitivity of each deduplication technique dimension to different hardware resource utilization rates, making it applicable to various I / O load conditions. The specific process of determining the deduplication strategy is explained in detail below with reference to specific embodiments:

[0061] In some embodiments, the deduplication timing, deduplication scope, block partitioning strategy, hash algorithm, and hot / cold data index are determined based on the utilization rates of each hardware resource and the corresponding hardware resource utilization thresholds, including:

[0062] S121: When the CPU utilization is less than the first CPU utilization threshold and the memory utilization is less than the first memory utilization threshold, the deduplication timing is determined to be online deduplication.

[0063] S122: When the CPU utilization is not less than the first CPU utilization threshold and / or the memory utilization is not less than the first memory utilization threshold, the deleting timing is determined to be background deleting.

[0064] Specifically, deduplication can be performed online or in the background. Online deduplication requires real-time hash calculation and hash fingerprint comparison, which increases CPU and memory overhead. Hash calculation and hash fingerprint comparison consume about 15%-30% of CPU computing power. When the fingerprint index resides in memory, about 1-3GB of memory space is required per TB of data, which is not significantly related to the interconnection bandwidth between nodes or the status of the backend disk.

[0065] Therefore, when CPU utilization is less than the first CPU utilization threshold and memory utilization is less than the first memory utilization threshold, online deduplication is determined to prioritize deduplication rate without affecting storage system performance. When CPU utilization is not less than the first CPU utilization threshold and / or memory utilization is not less than the first memory utilization threshold, background deduplication is determined to prioritize storage system performance.

[0066] S123: When the CPU utilization is less than the third CPU utilization threshold, the memory utilization is less than the first memory utilization threshold, the inter-node interconnect bandwidth utilization is less than the inter-node interconnect bandwidth utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold, the deduplication scope is determined to be global deduplication.

[0067] S124: When the CPU utilization is not less than the third CPU utilization threshold, and / or the memory utilization is not less than the first memory utilization threshold, and / or the inter-node interconnect bandwidth utilization is not less than the inter-node interconnect bandwidth utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold, the deduplication scope is determined to be local deduplication.

[0068] Secondly, deduplication can be categorized into global deduplication and local deduplication. For storage systems, global deduplication corresponds to a pool, while local deduplication corresponds to a volume (vdisk). Global deduplication achieves system-wide deduplication by synchronizing hash fingerprints across nodes, which increases CPU consumption. Furthermore, global deduplication requires storing more hash fingerprints, and each comparison of hash fingerprints or writing of deduplicated data necessitates a significant increase in cached data. It also requires data retrieval and transmission across nodes, placing certain demands on inter-node bandwidth and requiring the storage and retrieval of even more hash fingerprints.

[0069] Therefore, when CPU utilization is less than the third CPU utilization threshold, memory utilization is less than the first memory utilization threshold, inter-node interconnect bandwidth utilization is less than the inter-node interconnect bandwidth utilization threshold, and backend disk utilization is less than the backend disk utilization threshold, the deduplication scope is determined to be global deduplication. Global deduplication will greatly improve the deduplication rate, with actual measured deduplication rates reaching 65%-80%. When CPU utilization is not less than the third CPU utilization threshold, and / or memory utilization is not less than the first memory utilization threshold, and / or inter-node interconnect bandwidth utilization is not less than the inter-node interconnect bandwidth utilization threshold, and / or backend disk utilization is not less than the backend disk utilization threshold, the deduplication scope is determined to be local deduplication, thus prioritizing storage performance.

[0070] S125: When the CPU utilization is less than the second CPU utilization threshold and the memory utilization is less than the second memory utilization threshold, the block partitioning strategy is determined to be variable block partitioning.

[0071] S126: When the CPU utilization is not less than the second CPU utilization threshold and / or the memory utilization is not less than the second memory utilization threshold, the block partitioning strategy is determined to be fixed block partitioning.

[0072] Furthermore, the block partitioning strategy is divided into fixed partitioning and variable partitioning. Variable partitioning dynamically divides deduplication data blocks based on data content. It scans the data stream using a sliding window algorithm (such as Rabin fingerprinting), and triggers block splitting when preset conditions are met (such as hash value modulus matching), generating data blocks of non-fixed size (typical range: 4KB-64KB). Rabin fingerprinting calculation consumes 5%-10% of CPU resources. The memory consumption of variable partitioning mainly comes from sliding window state maintenance, hash fingerprint index storage, and parallel computing cache consumption, and is unrelated to inter-node interconnect bandwidth and backend disk utilization.

[0073] Therefore, when CPU utilization is less than the second CPU utilization threshold and memory utilization is less than the second memory utilization threshold, the block partitioning strategy is determined to be variable partitioning. When CPU utilization is not less than the second CPU utilization threshold and / or memory utilization is not less than the second memory utilization threshold, the block partitioning strategy is determined to be fixed partitioning, with a block size of 64K.

[0074] S127: When the CPU utilization is less than the second CPU utilization threshold, the memory utilization is less than the third memory utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold, the hash algorithm is determined to be the first hash algorithm.

[0075] S128: When the CPU utilization is not less than the second CPU utilization threshold, and / or the memory utilization is not less than the third memory utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold, the hash algorithm is determined to be the second hash algorithm.

[0076] The hash algorithms primarily used in storage systems are SHA-256 and xxHash64+xxHash3. SHA-256 is a computationally intensive algorithm, advantageous for its low collision rate and tamper resistance. xxHash64 is a non-encrypted hash algorithm designed for high speed and low collision rate; its core advantage lies in its extremely fast computation speed. SHA-256 requires calculating a 256-bit hash value for each deduplicated data block, placing high demands on CPU performance while consuming very little additional memory. SHA-256 requires an additional 32 bytes of storage per data block, while xxHash64 requires only 8 bytes per data block. Therefore, SHA-256 has a certain impact on backend disk performance.

[0077] Therefore, when CPU utilization is less than the second CPU utilization threshold, memory utilization is less than the third memory utilization threshold, and backend disk utilization is less than the backend disk utilization threshold, the hash algorithm is determined to be the first hash algorithm. When CPU utilization is not less than the second CPU utilization threshold, and / or memory utilization is not less than the third memory utilization threshold, and / or backend disk utilization is not less than the backend disk utilization threshold, the hash algorithm is determined to be the second hash algorithm.

[0078] It should also be noted that the collision rate of xxHash64 is (1 / 2). 32 When dealing with large amounts of data, deduplication errors can easily occur, leading to data inconsistency. Therefore, xxHash3 is used for secondary verification, and its actual processing performance is still higher than SHA-256. It should be noted that regardless of the hash algorithm used, a Bloom filter is needed to filter out data blocks not in the hash fingerprint table before calculating the hash value.

[0079] S129: When memory utilization is less than the first memory utilization threshold, add hot data index.

[0080] S130: Reduce hot data indexes when memory utilization is not less than the third memory utilization threshold.

[0081] Finally, the hot and cold data indexes are only related to the state of memory. The cold data index is the hash fingerprint value stored on disk, while the hot data index is the hash fingerprint value stored in the cache. Therefore, when memory utilization is less than the first memory utilization threshold, the hot data index is increased. When memory utilization is not less than the third memory utilization threshold, the hot data index is decreased.

[0082] It is worth noting that the first CPU utilization threshold is lower than the second CPU utilization threshold, and the second CPU utilization threshold is lower than the third CPU utilization threshold; similarly, the first memory utilization threshold is lower than the second memory utilization threshold, and the second memory utilization threshold is lower than the third memory utilization threshold. In this embodiment, there are no restrictions on the magnitudes of the first, second, and third CPU utilization thresholds, nor on the magnitudes of the first, second, and third memory utilization thresholds, nor on the magnitudes of the inter-node interconnect bandwidth utilization threshold and the backend disk utilization threshold. For example, the first CPU utilization threshold can be set to 40%, the second CPU utilization threshold to 50%, and the third CPU utilization threshold to 80%; the first memory utilization threshold to 50%, the second memory utilization threshold to 60%, and the third memory utilization threshold to 90%; the inter-node interconnect bandwidth utilization threshold to 60%, and the backend disk utilization threshold to 90%.

[0083] In summary, this solution, for different I / O load conditions, adaptively selects the optimal deduplication timing, deduplication range, deduplication blocks, and hash algorithms based on the utilization of CPU, memory, node interconnect bandwidth, and backend disks. Without sacrificing storage performance, it ensures the deduplication rate, saves storage space, and optimizes storage performance.

[0084] Based on the above embodiments, in some embodiments, after performing data deduplication according to the deduplication policy and input / output requests, the method further includes:

[0085] S16: Monitor hardware resource utilization according to the second preset period and return to the step of obtaining the hardware resource utilization threshold corresponding to each hardware resource utilization.

[0086] The first preset period is shorter than the second preset period.

[0087] In practice, after data deduplication is performed according to the deduplication policy and input / output requests, the adjustment of the deduplication policy will inevitably lead to changes in hardware resource utilization. These changes are proactive adjustments, and the resource monitoring module will not immediately report them; otherwise, the selection of deduplication features would become invalid, and it would also cause fluctuations in hardware resource utilization.

[0088] Therefore, to avoid the above situation, this embodiment specifically monitors hardware resource utilization according to a second preset period and returns to the step of obtaining the hardware resource utilization threshold corresponding to each hardware resource utilization rate. It should be noted that the first preset period is shorter than the second preset period. This embodiment does not impose a limit on the second preset period. For example, when the first preset period is 1 second, the second preset period can be set to 5 minutes. That is, after executing the data deduplication interval of 5 minutes, the hardware resources are monitored again, returning to the step of obtaining the hardware resource utilization threshold corresponding to each hardware resource utilization rate, and the data deduplication process is executed again, thereby avoiding fluctuations in hardware resource utilization.

[0089] Based on the above embodiments, in some embodiments, after performing data deduplication according to the deduplication policy and input / output requests, the method further includes:

[0090] S17: Determine the deduplication rate and system performance changes for this data deduplication.

[0091] S18: Based on the deduplication rate and system performance change information, determine whether the current data deduplication meets the preset conditions; if not, proceed to step S19; if yes, end.

[0092] S19: Reconfigure the hardware resource utilization thresholds corresponding to each hardware resource utilization rate, and return to the step of monitoring hardware resource utilization according to the first preset period.

[0093] After data deduplication is completed, the process can be evaluated to determine the effectiveness of the deduplication strategy. Specifically, the deduplication rate and system performance changes are first determined. The deduplication ratio measures the proportion of duplicate data reduced by the deduplication technique to the total amount of original data; it directly reflects the effectiveness of the deduplication strategy in eliminating redundant data. The system performance change information assesses the impact of the deduplication operation on the overall performance of the storage system, including I / O throughput, latency, and CPU and memory utilization.

[0094] Furthermore, based on the deduplication rate and system performance change information, it is determined whether the current data deduplication meets the preset conditions. It should be noted that this embodiment does not impose restrictions on the preset conditions. For example, a high deduplication rate indicates that the deduplication strategy is very effective and can significantly reduce storage space usage; a low deduplication rate may mean that there is less duplicate data in the dataset, or that the current deduplication strategy has failed to fully identify and eliminate redundant data. Therefore, a deduplication rate threshold can be set as part of the preset conditions. At the same time, while ensuring a high deduplication rate, the impact on system performance should be minimized. An ideal deduplication strategy should reduce storage space usage while minimizing negative impacts on system performance. In cases of significant performance degradation, it may be necessary to optimize the execution timing of the deduplication strategy or adjust system resource configuration to balance space optimization and performance requirements. Therefore, a system performance change threshold (including but not limited to I / O throughput threshold, latency threshold, or CPU and memory utilization threshold) can also be set as another part of the preset conditions.

[0095] Therefore, when it is confirmed that the current data deduplication does not meet the preset conditions, the hardware resource utilization thresholds corresponding to each hardware resource utilization rate are reconfigured, and the process returns to the step of monitoring hardware resource utilization according to the first preset cycle, thereby executing the next deduplication; when it is confirmed that the current data deduplication meets the preset conditions, the process ends. This achieves the evaluation of the current data deduplication, thereby accurately determining the effectiveness of the deduplication strategy.

[0096] In addition to the above, the effectiveness of a deduplication strategy can also be determined by the amount of storage space saved. Specifically, storage space saving refers to the actual reduction in storage space achieved through deduplication technology; this is the direct benefit of the deduplication strategy. A significant storage space saving indicates that the deduplication strategy effectively optimizes the utilization of storage resources and reduces storage costs, which is particularly important for large-scale data storage environments. Smaller storage space savings may require a reassessment of the deduplication strategy configuration, such as adjusting block size, hash algorithm, or deduplication range, to improve space optimization. Therefore, in practical implementation, after data deduplication is completed, it can be determined whether the storage space saving meets a preset value. If so, the data deduplication effect is considered good, and no strategy adjustment is needed; otherwise, the data deduplication strategy needs to be adjusted. This ensures high-performance execution of data deduplication.

[0097] To enable those skilled in the art to better understand this solution, the following is a detailed execution process of a data deduplication method:

[0098] When the real-time resource monitoring module detects that CPU utilization is 30%, memory utilization is 20%, inter-node interconnect bandwidth utilization is 30%, and backend disk utilization is 25%, it confirms that the current storage system is operating under low load. At this time, based on the deduplication feature selection function, the selected deduplication timing is online deduplication, the deduplication scope is global deduplication, the block partitioning strategy is variable block partitioning, the hash algorithm is SHA-256, and the cold and hot data indexing method is hot data indexing.

[0099] After data deduplication is completed based on the above deduplication strategy, CPU utilization, memory utilization, and inter-node interconnect bandwidth utilization will definitely increase. Suppose CPU utilization rises to 55%, memory utilization to 58%, and inter-node interconnect bandwidth utilization to 60%. These hardware resource utilization rates have exceeded the corresponding thresholds. However, at this point, the real-time resource monitoring module will not report to the multi-dimensional deduplication feature selection module; otherwise, the deduplication feature selection would fail, and it would also cause fluctuations in hardware resource utilization.

[0100] Based on this, after adjusting the multi-dimensional deduplication features, the real-time resource monitoring module will report hardware resource utilization again after 5 minutes. At this time, due to increased front-end I / O traffic, memory utilization becomes 65%, exceeding the corresponding second memory utilization threshold of 60%. The real-time resource monitoring module reports this to the multi-dimensional deduplication feature selection module. The multi-dimensional deduplication feature selection module detects that the memory change in the deduplication range has crossed the threshold and adjusts global deduplication to local deduplication. Continued increase in I / O traffic causes any one of the following—CPU utilization, memory utilization, inter-node interconnect bandwidth utilization, and back-end disk utilization—to reach 100%. Assuming memory utilization reaches 100%, a complete adjustment of the deduplication strategy will be triggered. The deduplication features will change to background deduplication, local deduplication, fixed block partitioning, xxHash64+xxHash3, and cold data indexing, respectively, resulting in a decrease in the utilization of various hardware resources. After 5 minutes, once a hardware resource utilization crosses the threshold, it will be reported again to the multi-dimensional deduplication feature selection module for a new round of deduplication strategy adjustments.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0102] Figure 3 This is a schematic diagram of a data deduplication device provided in an embodiment of the present invention. The device is applied to a storage system; such as... Figure 3 As shown, the device includes:

[0103] The monitoring module 10 is used to monitor the hardware resource utilization rate according to a first preset period when it receives an input / output request from the host side; wherein the hardware resource utilization rate includes at least the central processing unit utilization rate, memory utilization rate, inter-node interconnection bandwidth utilization rate, and back-end disk utilization rate.

[0104] The acquisition module 11 is used to acquire the hardware resource utilization threshold corresponding to the utilization rate of each hardware resource.

[0105] The deduplication module 12 is used to determine the deduplication strategy based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, and to perform data deduplication based on the deduplication strategy and input / output requests.

[0106] In some embodiments, it also includes:

[0107] The configuration submodule is used to configure the initial hardware resource utilization thresholds corresponding to the utilization rates of each hardware resource; wherein, the initial hardware resource utilization thresholds include at least the initial CPU utilization threshold, the initial memory utilization threshold, the initial inter-node interconnect bandwidth utilization threshold, and the initial back-end disk utilization threshold.

[0108] The first determination submodule is used to determine the overall status and hardware type of the system; wherein, the overall status includes at least single-controller cluster, multi-controller cluster, ordinary cluster and dual-active cluster; the hardware type includes at least the central processing unit model and back-end disk type;

[0109] The adjustment module is used to adjust the initial hardware resource utilization thresholds according to the overall status and hardware type to generate the hardware resource utilization thresholds.

[0110] In some embodiments, the adjustment module includes:

[0111] The first adjustment submodule is used to increase the initial inter-node interconnect bandwidth utilization threshold when the overall state is a single-control cluster.

[0112] The second adjustment submodule is used to reduce the initial CPU utilization threshold when the overall state is a dual-active cluster.

[0113] The third adjustment submodule is used to increase the initial CPU utilization threshold when the CPU model is the target model.

[0114] The fourth adjustment submodule is used to reduce the initial backend disk utilization threshold when the backend disk type is mechanical hard drive.

[0115] In some embodiments, the deduplication module 12 includes:

[0116] The deduplication strategy determination module is used to determine the deduplication timing, deduplication range, block partitioning strategy, hash algorithm, and hot and cold data index based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold.

[0117] Among them, the deduplication timing includes online deduplication and background deduplication; the deduplication scope includes global deduplication and local deduplication; the block division strategy includes fixed block division and variable block division; the hash algorithm includes a first hash algorithm and a second hash algorithm, with the collision rate and calculation speed of the first hash algorithm being lower than those of the second hash algorithm.

[0118] In some embodiments, the deduplication policy determination module includes:

[0119] The second determining submodule is used to determine the deduplication timing as online deduplication when the CPU utilization is less than the first CPU utilization threshold and the memory utilization is less than the first memory utilization threshold.

[0120] The third determining submodule is used to determine the deleting timing as background deleting when the CPU utilization is not less than the first CPU utilization threshold and / or the memory utilization is not less than the first memory utilization threshold.

[0121] The fourth determination submodule is used to determine the deduplication scope as global deduplication when the CPU utilization is less than the third CPU utilization threshold, the memory utilization is less than the first memory utilization threshold, the inter-node interconnect bandwidth utilization is less than the inter-node interconnect bandwidth utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold.

[0122] The fifth determination submodule is used to determine the deduplication range as local deduplication when the CPU utilization is not less than the third CPU utilization threshold, and / or the memory utilization is not less than the first memory utilization threshold, and / or the inter-node interconnect bandwidth utilization is not less than the inter-node interconnect bandwidth utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold.

[0123] The sixth determination submodule is used to determine the block partitioning strategy as variable block partitioning when the CPU utilization is less than the second CPU utilization threshold and the memory utilization is less than the second memory utilization threshold.

[0124] The seventh determination submodule is used to determine the block partitioning strategy as fixed block partitioning when the CPU utilization is not less than the second CPU utilization threshold and / or the memory utilization is not less than the second memory utilization threshold.

[0125] The eighth determination submodule is used to determine the hash algorithm as the first hash algorithm when the CPU utilization is less than the second CPU utilization threshold, the memory utilization is less than the third memory utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold.

[0126] The ninth determination submodule is used to determine the hash algorithm as the second hash algorithm when the CPU utilization is not less than the second CPU utilization threshold, and / or the memory utilization is not less than the third memory utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold.

[0127] The tenth determination submodule is used to add hot data indexes when memory utilization is less than the first memory utilization threshold;

[0128] The eleventh determination submodule is used to reduce hot data indexes when memory utilization is not less than the third memory utilization threshold;

[0129] Among them, the first CPU utilization threshold is less than the second CPU utilization threshold, the second CPU utilization threshold is less than the third CPU utilization threshold; the first memory utilization threshold is less than the second memory utilization threshold, and the second memory utilization threshold is less than the third memory utilization threshold.

[0130] In some embodiments, it also includes:

[0131] The first monitoring submodule is used to monitor the hardware resource utilization rate according to the second preset period and return to the step of obtaining the hardware resource utilization rate threshold corresponding to each hardware resource utilization rate.

[0132] The first preset period is shorter than the second preset period.

[0133] In some embodiments, it also includes:

[0134] The information determination module is used to determine the deduplication rate and system performance changes for this data deduplication.

[0135] The judgment module is used to determine whether the current data deduplication meets the preset conditions based on the deduplication rate and system performance change information. If not, the hardware resource utilization threshold corresponding to each hardware resource utilization rate is reconfigured, and the process returns to the step of monitoring hardware resource utilization according to the first preset cycle. If yes, the process ends.

[0136] For a description of the features in the embodiment corresponding to the data deduplication device, please refer to the relevant description in the embodiment corresponding to the data deduplication method, which will not be repeated here.

[0137] Embodiments of the present invention also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data deduplication method embodiments.

[0138] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data deduplication method embodiments when running.

[0139] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0140] Embodiments of the present invention also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data deduplication method embodiments.

[0141] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data deduplication method embodiments.

[0142] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0143] The data deduplication method, apparatus, device, and medium provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. It should be noted that those skilled in the art can make several improvements and modifications to the present invention without departing from the principles of the present invention, and these improvements and modifications also fall within the protection scope of the present invention.

Claims

1. A data deduplication method, characterized in that, Applied to a storage system; the method includes: When an input / output request is received from the host side, the hardware resource utilization rate is monitored according to a first preset period; wherein, the hardware resource utilization rate includes at least the central processing unit utilization rate, memory utilization rate, inter-node interconnect bandwidth utilization rate, and back-end disk utilization rate; Obtain the hardware resource utilization threshold corresponding to each of the aforementioned hardware resource utilization rates; A deduplication strategy is determined based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, and data deduplication is performed based on the deduplication strategy and the input / output request. The deduplication strategy is determined based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, including: Based on the utilization rates of each hardware resource and the corresponding hardware resource utilization thresholds, determine the deduplication timing, deduplication range, block partitioning strategy, hash algorithm, and hot / cold data index. The deduplication timing includes online deduplication and background deduplication; the deduplication scope includes global deduplication and local deduplication; the block partitioning strategy includes fixed block partitioning and variable block partitioning; the hash algorithm includes a first hash algorithm and a second hash algorithm, wherein the collision rate and calculation speed of the first hash algorithm are lower than those of the second hash algorithm, respectively. Based on the utilization rates of the aforementioned hardware resources and the corresponding hardware resource utilization thresholds, the deduplication timing, deduplication range, block partitioning strategy, hash algorithm, and hot / cold data index are determined, including: When the CPU utilization is less than a first CPU utilization threshold and the memory utilization is less than a first memory utilization threshold, the deduplication timing is determined to be online deduplication. When the CPU utilization is not less than the first CPU utilization threshold and / or the memory utilization is not less than the first memory utilization threshold, the deduplication timing is determined to be background deduplication. When the CPU utilization is less than the third CPU utilization threshold, the memory utilization is less than the first memory utilization threshold, the inter-node interconnect bandwidth utilization is less than the inter-node interconnect bandwidth utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold, the deduplication range is determined to be the global deduplication. When the CPU utilization is not less than the third CPU utilization threshold, and / or the memory utilization is not less than the first memory utilization threshold, and / or the inter-node interconnect bandwidth utilization is not less than the inter-node interconnect bandwidth utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold, the deduplication range is determined to be the local deduplication. When the CPU utilization is less than the second CPU utilization threshold and the memory utilization is less than the second memory utilization threshold, the block partitioning strategy is determined to be the variable block partitioning. When the CPU utilization is not less than the second CPU utilization threshold, and / or the memory utilization is not less than the second memory utilization threshold, the block partitioning strategy is determined to be the fixed block partitioning. When the CPU utilization is less than the second CPU utilization threshold, the memory utilization is less than the third memory utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold, the hash algorithm is determined to be the first hash algorithm. When the CPU utilization is not less than the second CPU utilization threshold, and / or the memory utilization is not less than the third memory utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold, the hash algorithm is determined to be the second hash algorithm. When the memory utilization rate is less than the first memory utilization threshold, add a hot data index; When the memory utilization rate is not less than the third memory utilization threshold, reduce the hot data index; Among them, the first CPU utilization threshold is less than the second CPU utilization threshold, the second CPU utilization threshold is less than the third CPU utilization threshold; the first memory utilization threshold is less than the second memory utilization threshold, and the second memory utilization threshold is less than the third memory utilization threshold.

2. The data deduplication method according to claim 1, characterized in that, Before monitoring the hardware resource utilization rate within the system according to the first preset cycle, the following is also included: Configure initial hardware resource utilization thresholds corresponding to the utilization rates of each of the aforementioned hardware resources; wherein, the initial hardware resource utilization thresholds include at least an initial CPU utilization threshold, an initial memory utilization threshold, an initial inter-node interconnect bandwidth utilization threshold, and an initial back-end disk utilization threshold; Determine the overall status and hardware type of the system; wherein, the overall status includes at least single-controller cluster, multi-controller cluster, ordinary cluster, and dual-active cluster; the hardware type includes at least the CPU model and back-end disk type; Adjust each of the initial hardware resource utilization thresholds according to the overall state and the hardware type to generate each of the hardware resource utilization thresholds.

3. The data deduplication method according to claim 2, characterized in that, Adjusting the initial hardware resource utilization thresholds based on the overall state and the hardware type includes: When the overall state is the single-control cluster, increase the initial inter-node interconnection bandwidth utilization threshold. When the overall state is the dual-active cluster, reduce the initial CPU utilization threshold; When the CPU model is the target model, increase the initial CPU utilization threshold; When the backend disk type is a mechanical hard drive, reduce the initial backend disk utilization threshold.

4. The data deduplication method according to any one of claims 1 to 3, characterized in that, After performing data deduplication according to the deduplication policy and the input / output request, the process further includes: The hardware resource utilization rate is monitored according to the second preset period, and the process returns to the step of obtaining the hardware resource utilization rate threshold corresponding to each hardware resource utilization rate. Wherein, the first preset period is shorter than the second preset period.

5. The data deduplication method according to claim 4, characterized in that, After performing data deduplication according to the deduplication policy and the input / output request, the process further includes: Determine the deduplication rate and system performance changes for this data deduplication process; Based on the deduplication rate and the system performance change information, determine whether the current data deduplication meets the preset conditions; If not, then reconfigure the hardware resource utilization threshold corresponding to each of the aforementioned hardware resource utilization rates, and return to the step of monitoring hardware resource utilization according to the first preset period; If so, then the process ends.

6. A data deduplication device, characterized in that, Applied to a storage system; the device includes: The monitoring module is used to monitor hardware resource utilization according to a first preset period when it receives an input / output request from the host side; wherein, the hardware resource utilization includes at least the central processing unit utilization, memory utilization, inter-node interconnect bandwidth utilization, and back-end disk utilization. The acquisition module is used to acquire the hardware resource utilization threshold corresponding to each of the hardware resource utilization rates. The deduplication module is used to determine a deduplication strategy based on the utilization rate of each hardware resource and the corresponding hardware resource utilization threshold, and to perform data deduplication based on the deduplication strategy and the input / output request. The deduplication module includes: The deduplication strategy determination module is used to determine the deduplication timing, deduplication scope, block partitioning strategy, hash algorithm, and hot / cold data index based on the utilization rates of each hardware resource and the corresponding hardware resource utilization thresholds. The deduplication timing includes online deduplication and background deduplication; the deduplication scope includes global deduplication and local deduplication; the block partitioning strategy includes fixed block partitioning and variable block partitioning; and the hash algorithm includes a first hash algorithm and a second hash algorithm, wherein the collision rate and computation speed of the first hash algorithm are lower than those of the second hash algorithm, respectively. The deduplication strategy determination module includes: The second determining submodule is used to determine the online deduplication timing when the CPU utilization is less than the first CPU utilization threshold and the memory utilization is less than the first memory utilization threshold. The third determining submodule is used to determine the deduplication timing as background deduplication when the CPU utilization is not less than the first CPU utilization threshold and / or the memory utilization is not less than the first memory utilization threshold. The fourth determining submodule is used to determine the deduplication range as the global deduplication when the CPU utilization is less than the third CPU utilization threshold, the memory utilization is less than the first memory utilization threshold, the inter-node interconnect bandwidth utilization is less than the inter-node interconnect bandwidth utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold. The fifth determining submodule is used to determine the deduplication range as the local deduplication when the CPU utilization is not less than the third CPU utilization threshold, and / or the memory utilization is not less than the first memory utilization threshold, and / or the inter-node interconnect bandwidth utilization is not less than the inter-node interconnect bandwidth utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold. The sixth determining submodule is used to determine the block partitioning strategy as the variable block partitioning when the CPU utilization is less than the second CPU utilization threshold and the memory utilization is less than the second memory utilization threshold. The seventh determining submodule is used to determine the block partitioning strategy as the fixed partitioning when the CPU utilization is not less than the second CPU utilization threshold and / or the memory utilization is not less than the second memory utilization threshold. The eighth determining submodule is used to determine the hash algorithm as the first hash algorithm when the CPU utilization is less than the second CPU utilization threshold, the memory utilization is less than the third memory utilization threshold, and the back-end disk utilization is less than the back-end disk utilization threshold. The ninth determining submodule is used to determine the hash algorithm as the second hash algorithm when the CPU utilization is not less than the second CPU utilization threshold, and / or the memory utilization is not less than the third memory utilization threshold, and / or the back-end disk utilization is not less than the back-end disk utilization threshold. The tenth determining submodule is used to add a hot data index when the memory utilization rate is less than the first memory utilization rate threshold; The eleventh determination submodule is used to reduce hot data indexes when the memory utilization rate is not less than the third memory utilization rate threshold. Among them, the first CPU utilization threshold is less than the second CPU utilization threshold, the second CPU utilization threshold is less than the third CPU utilization threshold; the first memory utilization threshold is less than the second memory utilization threshold, and the second memory utilization threshold is less than the third memory utilization threshold.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data deduplication method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data deduplication method as described in any one of claims 1 to 5.

Images