Data deduplication method and related system

Abstract

Provided in the present application is a data deduplication method. The method comprises: receiving a write request; writing a first data block in the write request into a storage device; writing metadata of the first data block, such as a fingerprint and a logical address, into a first partition, which is determined according to a feature of the first data block, among a plurality of partitions of a metadata management structure; when a fingerprint, which is the same as the fingerprint of the first data block, is present in the first partition, deleting the metadata of the first data block from the first partition; and according to address information of the first data block, deleting the first data block from the storage device. In the method, a metadata management structure is actively partitioned, and metadata of a data block is written into a partition corresponding to a feature of the data block, such that the problem of unfrequently updated data being eliminated because resources of the unfrequently updated data are occupied by frequently updated data, thereby increasing the deduplication rate.

Description

A data deduplication method and related system

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 24, 2022, with application number 202210730080.0 and invention name “A Data Deduplication Method”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on September 16, 2022, with application number 202211132110.4 and invention name “A Data Deduplication Method and Related System”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data deduplication method, device, storage system, computer-readable storage medium, and computer program product. Background Art

[0003] With the development of the computing industry, the value of data is being fully realized, and the scale of data centers has grown from petabytes (PB) to zettabytes (ZB). This massive amount of data stored in data centers contains a significant amount of redundant data. Statistics show that in two major application scenarios, primary storage systems and backup storage systems, data redundancy rates are approximately 50% and 85%, respectively. How to effectively reduce redundant data and, thereby, lower storage costs has become a hot research topic.

[0004] The industry generally uses data deduplication (DD) to reduce data redundancy. Data deduplication, also known as deduplication, works by breaking data into blocks and calculating fingerprints based on their content. By comparing the fingerprints of different blocks, DD identifies and deletes blocks with duplicate content, eliminating data redundancy.

[0005] Data block fingerprints are typically written to log files in an append-only manner. During deduplication, the fingerprints in the log files are sorted manually or periodically, and the sorted fingerprints are merged with the fingerprints in the fingerprint file. Data blocks with duplicate content are then deleted based on the merged result.

[0006] Data stored in data centers can be divided into frequently updated data and infrequently updated data based on update frequency. However, the proportion of frequently updated data is typically higher than that of infrequently updated data. Frequently updated data is often difficult to deduplicate, meaning that the proportion of this data is higher. This leads to resource congestion, while infrequently updated data receives a lower percentage of allocated space, making it more likely to be eliminated, resulting in a loss of deduplication efficiency.

[0007] Summary of the Invention

[0008] This application provides a data deduplication method that actively partitions a metadata management structure and writes metadata, such as fingerprints and address information, of data blocks to partitions corresponding to the data block's characteristics. This method prevents infrequently updated data from being squeezed out by frequently updated data and subsequently eliminated, thereby improving the deduplication rate. This application also provides a device, storage system, computer-readable storage medium, and computer program product corresponding to the above method.

[0009] In a first aspect, the present application provides a data deduplication method. The method can be applied to a storage system, including a centralized storage system or a distributed storage system. Among them, the centralized storage system can also be divided into a centralized storage system with integrated disk and controller or a centralized storage system with separated disk and controller, and the distributed storage system can also be divided into a distributed storage system with integrated storage and computing or a distributed storage system with separated storage and computing. The centralized storage system has an engine, which includes a controller, and the controller may include a processor and a memory. The processor can load the program code in the memory to execute the data deduplication method of the present application. Similarly, the distributed storage system includes computing nodes and storage nodes, and the computing nodes include a processor and a memory. The processor can load the program code in the memory to execute the data deduplication method of the present application.

[0010] Specifically, a storage system receives a write request that includes a first data block. The storage system then writes the first data block to a storage device (e.g., a hard disk) and then writes metadata for the first data block to a first partition of a plurality of partitions in a metadata management structure. The first partition is determined based on characteristics of the first data block. The metadata for the first data block includes a fingerprint and address information of the first data block. When a fingerprint identical to the fingerprint of the first data block exists in the first partition, the storage system deletes the metadata for the first data block in the first partition and deletes the first data block from the storage device based on the address information of the first data block.

[0011] In this method, metadata of data blocks with different characteristics can be written into different partitions of the metadata management structure. For example, metadata of data blocks that are frequently updated can be written into a partition with a smaller capacity, and metadata of data blocks that are infrequently updated can be written into a partition with a larger capacity. This prevents data that is infrequently updated from being squeezed out by data that is frequently updated and then eliminated, thereby improving the deduplication rate.

[0012] In some possible implementations, the characteristic of the first data block is the fingerprint of the first data block. It should be noted that different data blocks may correspond to the same fingerprint. For example, when performing a backup, there may be multiple data blocks corresponding to the same fingerprint. When the storage system writes the metadata of the first data block into the metadata management structure, it may first determine the popularity of the fingerprint corresponding to the first data block, determine the first partition of the multiple partitions of the metadata management structure corresponding to the popularity based on the popularity of the fingerprint corresponding to the first data block, and then write the metadata of the first data block into the first partition.

[0013] In this method, the storage system determines the popularity of the fingerprint corresponding to the first data block and writes the first data block into the corresponding first partition based on the popularity. This can prevent infrequently updated data from being squeezed out by frequently updated data and then being eliminated, thereby improving the deduplication rate.

[0014] In some possible implementations, the address information of the first data block includes a logical address of the first data block. Accordingly, the storage system may further update the heat of the fingerprint corresponding to the first data block after writing the metadata of the first data block to the first partition. The storage system may determine the heat of the logical address of the first data block and add the heat of the logical address to the heat of the fingerprint corresponding to the first data block to update the heat of the fingerprint corresponding to the first data block.

[0015] In this method, the heat of the fingerprint is updated by accumulating the heat of the logical address to the heat of the fingerprint corresponding to the data block, which can provide a reference for subsequent metadata writing.

[0016] In some possible implementations, the write request may further include a second data block, and the popularity of the fingerprint corresponding to the second data block may be higher than the popularity of the fingerprint corresponding to the first data block. Accordingly, the storage system may further write the second data block to the storage device and write the metadata of the second data block to a second partition among the multiple partitions of the metadata management structure. The capacity of the second partition is smaller than that of the first partition.

[0017] This method writes data blocks with different fingerprint popularity into different partitions of the metadata management structure, thereby preventing the metadata of low-popularity data blocks from being squeezed out by the metadata of high-popularity data blocks and then being eliminated, thereby improving the deduplication rate.

[0018] In some possible implementations, the write request may further include a third data block, where the fingerprint of the third data block is the same as the fingerprint of the first data block. When writing metadata for the first data block, if the popularity of the fingerprint corresponding to the first data block is less than a preset popularity, and when writing metadata for the third data block, if the popularity of the fingerprint corresponding to the third data block is greater than the preset popularity, the storage system may write the metadata for the third data block to the second partition and move the metadata of data blocks in the first partition that have the same fingerprint as the third data block to the second partition.

[0019] In this way, the storage location of the metadata can be adjusted as data blocks are continuously written. For example, the metadata of data blocks with higher fingerprint popularity can be moved to the second partition, thereby reserving storage space in the first partition for the metadata of data blocks with lower fingerprint popularity, and storing metadata of data blocks with the same fingerprint in the second partition to support deduplication triggered by the second partition, thereby further improving the deduplication rate.

[0020] In some possible implementations, the storage system may also write the metadata of the third data block into the second partition, and eliminate the metadata of the data block in the first partition that has the same fingerprint as the third data block without moving it to the second partition. On the one hand, this can reduce movement overhead, and on the other hand, it can reserve storage space in the first partition for the metadata of the data blocks with lower fingerprint popularity, further improving the deduplication rate.

[0021] In some possible implementations, the capacities of the multiple partitions of the metadata management structure are determined based on a partition decision model. The partition decision model is configured to predict the corresponding partition benefit of each of the preset partition capacity combinations after application to the metadata management structure, and to determine the partition capacity combination with the maximum partition benefit as the capacity of the multiple partitions of the metadata management structure. The partition benefit is determined based on at least one of a deduplication rate and a partition adjustment cost.

[0022] This method constructs a partition decision model and actively partitions the metadata management structure through the partition decision model, thereby avoiding the frequent update data crowding out the resources of the infrequently updated data, and further avoiding the infrequently updated data from being eliminated, resulting in a loss of deduplication rate.

[0023] In some possible implementations, the partition benefit may be a deduplication rate, and the preset partition capacity combination may include a first capacity combination. The partition decision model may estimate the data hit rate to predict the corresponding deduplication rate after the partition capacity combination is applied to the data management structure.

[0024] Specifically, the partition decision model predicts the corresponding deduplication rate after the first partition capacity combination is applied to the metadata management structure in the following manner: obtaining the workload characteristics corresponding to each partition in the multiple partitions formed by applying the first partition capacity combination to the metadata management structure, obtaining the data distribution corresponding to each partition based on the workload characteristics corresponding to each partition, and obtaining the deduplication rate based on the data distribution corresponding to each partition and the capacity of each partition.

[0025] This method fits the data distribution of each partition based on the workload characteristics of each partition. Based on the data distribution and partition capacity of each partition, the hit rate can be predicted, and then the corresponding deduplication rate after applying the partition capacity combination can be predicted. Without actually running the storage system, the partition capacity combination with the highest deduplication rate can be predicted at a low cost to meet business needs.

[0026] In some possible implementations, partition capacity can be adjusted to account for potential workload changes. For example, the storage system can periodically adjust partition capacity. During project implementation, partition adjustments require operations such as reinitialization, incurring partition adjustment costs. The partition decision model can predict the partition adjustment costs based on the partition capacity ratio before and after the adjustment. The partition decision model can also predict partition revenue based on the yield rate and the partition adjustment cost. For example, the partition decision model can use the difference between the predicted yield rate and the predicted partition adjustment cost as the predicted partition revenue.

[0027] By reconstructing partition benefits, this method can make the evaluation of partition benefits more accurate and reasonable. With the goal of maximizing the reconstructed partition benefits, the determined partition capacity combination is more valuable for reference and can achieve a balance between the deduplication rate and the partition adjustment cost.

[0028] In some possible implementations, the storage system can periodically adjust the capacity of multiple partitions of the metadata management structure. When the adjustment time is reached, it determines whether to adjust the capacity of multiple partitions based on the partition income corresponding to the period before the adjustment time, the partition capacity combination, or the workload characteristics corresponding to each partition.

[0029] This method decides whether to adjust the capacity of a partition by using the feedback information corresponding to the period before the adjustment moment, such as the partition benefit, partition capacity combination, or the workload characteristics corresponding to each partition. This allows the partition to be flexibly adjusted based on changes in workload, ensuring as good a partition benefit as possible at different stages.

[0030] In some possible implementations, the storage system may store a fingerprint identical to the fingerprint of the first data block in the first partition, and when the number of fingerprints identical to the fingerprint of the first data block in the first partition reaches a preset threshold, the metadata of the first data block is deleted, and the first data block is deleted from the storage device based on the address information of the first data block.

[0031] The preset threshold value can be set based on an empirical value. For example, the preset threshold value can be set to 1. If there is a fingerprint identical to the first data block in the first partition, the storage system will delete the metadata of the first data block and the first data block. For another example, the preset threshold value can be set to 2. If there are two data blocks in the first partition whose fingerprints are identical to the fingerprint of the first data block, the storage system will delete the metadata of the first data block and the first data block. Furthermore, the storage system will retain one of the two data blocks with the same fingerprint as the first data block and its metadata, and delete the other data block and its metadata.

[0032] When the preset threshold is set to a smaller value, redundant data blocks and metadata can be deleted in a timely manner. When the preset threshold is set to a larger value, the number of deduplications can be reduced, avoiding frequent deduplication that takes up a large amount of resources and affects the normal operation of the business.

[0033] In some possible implementations, the address information of the first data block is the logical address of the first data block. The storage system may further write the logical address and physical address of the first data block into an address mapping table. Accordingly, when deleting the first data block, the storage system may obtain the physical address of the first data block from the address mapping table based on the logical address of the first data block, then locate the first data block from the storage device based on the physical address, and delete the first data block.

[0034] In this method, the storage system directly locates the first data block based on a single-hop mapping from a logical address to a physical address in an address mapping table, thereby shortening search time and improving deduplication efficiency.

[0035] In some possible implementations, after deleting the metadata of the first data block in the first partition, the storage system may further modify the physical address of the first data block in the forward mapping table to the fingerprint of the first data block.

[0036] The method can realize the relocation of the deleted data block, so that the physical address of the data block with the same fingerprint can be found based on the fingerprint later, and the data block can be accessed by accessing the physical address.

[0037] In some possible implementations, the storage system may further eliminate the metadata in at least one partition in the inverse mapping table when the at least one partition meets an elimination condition.

[0038] This method eliminates metadata from the inverse mapping table to reduce the size of metadata, thereby reducing memory overhead and ensuring system performance.

[0039] In a second aspect, the present application provides a data deduplication device. The device includes:

[0040] A communication module, configured to receive a write request, wherein the write request includes a first data block;

[0041] a data writing module, configured to write the first data block into a storage device;

[0042] The data writing module is further configured to write metadata of the first data block into a first partition among multiple partitions of a metadata management structure, where the first partition is determined based on characteristics of the first data block, and the metadata of the first data block includes a fingerprint and address information of the first data block;

[0043] A deduplication module is configured to delete metadata of the first data block in the first partition when a fingerprint identical to the fingerprint of the first data block exists in the first partition, and to delete the first data block from the storage device according to address information of the first data block.

[0044] In some possible implementations, the feature of the first data block is a fingerprint corresponding to the first data block, and the data writing module is specifically configured to:

[0045] Determining the popularity of the fingerprint corresponding to the first data block;

[0046] determining, according to the popularity of the fingerprint corresponding to the first data block, the first partition among the multiple partitions of the metadata management structure corresponding to the popularity;

[0047] Writing metadata of the first data block into the first partition.

[0048] In some possible implementations, the data writing module is further configured to:

[0049] After writing the metadata of the first data block into the first partition, determining the heat of the logical address of the first data block;

[0050] The heat of the logical address is accumulated to the heat of the fingerprint corresponding to the first data block to update the heat of the fingerprint corresponding to the first data block.

[0051] In some possible implementations, the write request further includes a second data block, the popularity of the fingerprint corresponding to the second data block is higher than the popularity of the fingerprint corresponding to the first data block, and the data writing module is further configured to:

[0052] The second data block is written into the storage device, and metadata of the second data block is written into a second partition among the multiple partitions of the metadata management structure, where the capacity of the second partition is smaller than the capacity of the first partition.

[0053] In some possible implementations, the capacities of the multiple partitions of the metadata management structure are determined based on a partition decision model, which is used to predict the corresponding partition benefits of each partition capacity combination in a preset partition capacity combination after it is applied to the metadata management structure, and to determine the partition capacity combination with the largest partition benefit as the capacity of the multiple partitions of the metadata management structure, wherein the partition benefit is determined based on at least one of a deduplication rate and a partition adjustment cost.

[0054] In some possible implementations, the partition benefit is a deduplication rate, the preset partition capacity combination includes a first partition capacity combination, and the partition decision model predicts the deduplication rate corresponding to the first partition capacity combination after being applied to the metadata management structure in the following manner:

[0055] Obtaining workload characteristics corresponding to each of the plurality of partitions formed by applying the first partition capacity combination to the metadata management structure;

[0056] Obtaining data distribution corresponding to each partition according to workload characteristics corresponding to each partition;

[0057] The deduplication rate is obtained according to the data distribution corresponding to each partition and the capacity of each partition.

[0058] In some possible implementations, the apparatus further includes a partitioning module, where the partitioning module is configured to:

[0059] periodically adjusting the capacity of multiple partitions of the metadata management structure;

[0060] When the adjustment time is reached, it is determined whether to adjust the capacities of the plurality of partitions according to the partition revenue corresponding to the period before the adjustment time, the partition capacity combination, or the workload characteristics corresponding to each partition.

[0061] In some possible implementations, the deduplication module is specifically configured to:

[0062] When a fingerprint identical to the fingerprint of the first data block exists in the first partition and the number of the fingerprints in the first partition reaches a preset threshold, the metadata of the first data block is deleted, and the first data block is deleted from the storage device according to the address information of the first data block.

[0063] In some possible implementations, the address information of the first data block is a logical address of the first data block, and the data writing module is further configured to:

[0064] Writing the logical address and the physical address of the first data block into the address mapping table;

[0065] The deduplication module is specifically used for:

[0066] Acquire the physical address of the first data block from the address mapping table according to the logical address of the first data block;

[0067] The first data block is found from the storage device according to the physical address, and the first data block is deleted.

[0068] In some possible implementations, the deduplication module is further configured to:

[0069] After deleting the metadata of the first data block in the first partition, the physical address of the first data block in the address mapping table is modified to the fingerprint of the first data block.

[0070] In some possible implementations, the apparatus further includes:

[0071] The elimination module is configured to eliminate the metadata in at least one partition in the inverse mapping table when the at least one partition meets an elimination condition.

[0072] In a third aspect, the present application provides a computer cluster. The computer cluster includes at least one computer, each of which includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, causing the computer or computer cluster to perform the data deduplication method described in the first aspect or any implementation of the first aspect.

[0073] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computer or a computer cluster to execute the data deduplication method described in the first aspect or any implementation of the first aspect.

[0074] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computer or a computer cluster, enables the computer or computer cluster to execute the data deduplication method described in the first aspect or any one of the implementations of the first aspect.

[0075] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0077] FIG1 is a system architecture diagram of a centralized storage system provided in an embodiment of the present application;

[0078] FIG2 is a system architecture diagram of a distributed storage system provided in an embodiment of the present application;

[0079] FIG3 is a schematic diagram of an embodiment of the present application providing a method of appending to a log file for metadata management;

[0080] FIG4 is a schematic diagram of metadata management through log files and inverse mapping tables provided in an embodiment of the present application;

[0081] FIG5 is a flow chart of a data deduplication method provided in an embodiment of the present application;

[0082] FIG6 is a flow chart of a data deduplication method provided in an embodiment of the present application;

[0083] FIG7 is a schematic diagram of a system resource feature extraction method provided in an embodiment of the present application;

[0084] FIG8 is a schematic diagram of a feature merging method provided in an embodiment of the present application;

[0085] FIG9 is a schematic diagram of obtaining structured features according to an embodiment of the present application;

[0086] FIG10 is a schematic diagram of a process of partition decision modeling provided in an embodiment of the present application;

[0087] FIG11 is a schematic diagram of a process for selecting an evaluation strategy provided in an embodiment of the present application;

[0088] FIG12 is a schematic diagram of an application scenario of a data deduplication method provided in an embodiment of the present application;

[0089] FIG13 is a flow chart of a data deduplication method provided in an embodiment of the present application applied to a global cache;

[0090] FIG14 is a schematic structural diagram of a data deduplication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0091] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0092] First, some technical terms involved in the embodiments of this application are introduced.

[0093] Data deduplication (DD), also known as deduplication, is a data reduction solution that removes duplicate data, reducing the storage space by storing only one copy of the same data on the storage medium. Deduplication works by partitioning data into blocks and calculating a block fingerprint (FP) based on the block's content. By comparing the fingerprints of different blocks, duplicate content can be identified and deleted, eliminating data redundancy.

[0094] A fingerprint is information used to identify a data block, determined based on its content. A data block's fingerprint can be a message digest calculated from the data block's content using a message digest algorithm. Message digest algorithms are typically implemented using hash functions, so a data block's fingerprint can also be a hash value or hash value determined using a hash function.

[0095] Deduplication can also be categorized by execution time. For example, deduplication can include pre-deduplication and post-deduplication. Pre-deduplication refers to deduplication before data is written to a storage medium (referred to as storage, such as a hard drive or other device). Pre-deduplication is also called online deduplication. Post-deduplication refers to deduplication after data is written to a storage medium (such as a hard drive or other device). Post-deduplication is also called background deduplication or offline deduplication.

[0096] The data deduplication method provided in the embodiments of the present application can be applied to different application scenarios, for example, it can be applied to a centralized storage system or a distributed storage system.

[0097] A centralized storage system consists of one or more master devices forming a central node, where data is centrally stored and all data processing for the entire system is centrally deployed. In other words, in a centralized storage system, terminals or clients are only responsible for data input and output, while data storage and control processing are completely handled by the central node. The most significant feature of a centralized system is its simple deployment structure, eliminating the need to consider how to deploy services across multiple nodes or the issues of distributed collaboration between these nodes.

[0098] The centralized storage system can include a disk-controller integrated centralized storage system or a disk-controller separated centralized storage system. Disk-controller integrated means that the storage medium (such as a hard disk) and the controller are integrated, while disk-controller separated means that the storage medium and the controller are separated.

[0099] Figure 1 is a system architecture diagram of a centralized storage system used in an embodiment of the present application. In the application scenario shown in Figure 1, users access data through applications. The computers that run these applications are called "application servers." Application server 100 can be a physical machine or a virtual machine created by virtualizing a physical machine. Physical machines include but are not limited to desktop computers, servers, laptops, and mobile devices.

[0100] Application servers access the storage system to store and retrieve data through Fibre Channel switch 110. However, switch 110 is an optional device; application servers 100 can also communicate directly with storage system 120 over the network. Alternatively, Fibre Channel switch 110 can be replaced with an Ethernet switch, an InfiniBand (IB) switch, or a Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) switch.

[0101] The storage system 120 shown in Figure 1 is a centralized storage system. A centralized storage system is characterized by a unified entry point through which all data from external devices must pass. This entry point is the centralized storage system's engine 121. Engine 121 is the core component of the centralized storage system, implementing many of the system's advanced functions.

[0102] As shown in Figure 1, there are one or more controllers in the engine 121. Figure 1 uses the example of an engine containing two controllers for explanation. There is a mirror channel between controller 0 and controller 1. Then, after controller 0 writes a copy of data into its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 stores the copy in its own local memory 124. In this way, controller 0 and controller 1 back up each other. When controller 0 fails, controller 1 can take over the business of controller 0. When controller 1 fails, controller 0 can take over the business of controller 1, thereby avoiding the unavailability of the entire storage system 120 due to hardware failure. When there are 4 controllers deployed in the engine 121, there is a mirror channel between any two controllers, so any two controllers back up each other.

[0103] The engine 121 also includes a front-end interface 125 and a back-end interface 126. The front-end interface 125 is used to communicate with the application server 100, thereby providing storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the storage system capacity. Through the back-end interface 126, the engine 121 can connect to more hard disks 134, thereby forming a very large storage resource pool.

[0104] In terms of hardware, as shown in FIG1 , the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) that is used to process data access requests (such as read requests or write requests) from outside the storage system (servers or other storage systems), and is also used to process requests generated within the storage system. For example, when the processor 123 receives write requests sent by the application server 100 through the front-end port 125, it temporarily stores the data in these write requests in the memory 124. When the total amount of data in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 through the back-end port for persistent storage.

[0105] Memory 124 refers to internal storage that directly exchanges data with the processor. It can read and write data at any time and at a high speed, serving as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory. For example, memory can be either random access memory (RAM) or read-only memory (ROM). For example, ROM is dynamic random access memory (DRAM) or storage class memory (SCM). DRAM is a type of semiconductor memory that, like most random access memory (RAM), is a volatile memory device. SCM is a hybrid storage technology that combines the characteristics of traditional storage devices and memory. Storage class memory can provide faster read and write speeds than hard drives, but has slower access speeds than DRAM and is also less expensive than DRAM. However, DRAM and SCM are merely exemplary in this embodiment; memory can also include other types of random access memory, such as static random access memory (SRAM). As for the read-only memory, for example, it can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc. In addition, the memory 124 can also be a dual in-line memory module or a dual-line memory module (DIMM), that is, a module composed of dynamic random access memory (DRAM), or a solid state disk (SSD). In actual applications, multiple memories 124 and different types of memories 124 can be configured in the controller 0. This embodiment does not limit the number and type of memory 113. In addition, the memory 124 can be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 124 will not be lost. A memory with a power-saving function is called a non-volatile memory.

[0106] Memory 124 stores software programs, and processor 123 executes the software programs in memory 124 to manage the hard disks. For example, the hard disks can be abstracted into a storage resource pool, which is then divided into logical unit number devices (LUNs) for server use. The LUNs here are essentially the hard disks seen on the server. Of course, some centralized storage systems are also file servers, providing shared file services for servers.

[0107] The hardware components and software structure of controller 1 (and other controllers not shown in FIG. 1 ) are similar to those of controller 0 and are not described in detail here.

[0108] Figure 1 shows a centralized storage system with separated disk and controller. In this system, the engine 121 may not have a hard disk slot, and the hard disk 134 may be placed in the hard disk frame 130. Depending on the type of communication protocol between the engine 121 and the hard disk frame 130, the hard disk frame 130 may be a serial attached small computer system interface (SAS) hard disk frame, or a non-volatile memory express (NVMe) hard disk frame, an Internet Protocol (IP) hard disk frame, or other types of hard disk frames. The SAS hard disk frame uses the SAS3.0 protocol, and each frame supports 25 SAS hard disks. The engine 121 is connected to the hard disk frame 130 via an onboard SAS interface or a SAS interface module. The NVMe hard disk frame is more like a complete computer system, with NVMe hard disks inserted in the NVMe hard disk frame. The NVMe hard disk frame is then connected to the engine 121 via an RDMA port.

[0109] The back-end interface 126 communicates with the disk enclosure 130. The back-end interface 126 exists in the form of an adapter card in the engine 121. One engine 121 can use two or more back-end interfaces 126 to connect to multiple disk enclosures 130. Alternatively, the adapter card can be integrated into the motherboard, in which case the adapter card can communicate with the processor 112 via the PCIE bus.

[0110] It should be noted that FIG1 only shows one engine 121 . However, in actual applications, the storage system may include two or more engines 121 , and redundancy or load balancing may be performed between the multiple engines 121 .

[0111] The storage system 120 shown in Figure 1 is a storage system with separate disks and controllers. In some possible implementations, the centralized storage system can also be a storage system with integrated disks and controllers. In a storage system with integrated disks and controllers, the engine 121 can have hard drive slots, and hard drives 134 can be directly deployed in the engine 121. The back-end interface 126 is optional. When the system's storage space is insufficient, more hard drives or disk enclosures can be connected through the back-end interface 126.

[0112] A distributed storage system stores data across multiple independent storage nodes. This system utilizes a scalable architecture, sharing the storage load across multiple storage nodes. This not only improves system reliability, availability, and access efficiency, but also facilitates scalability.

[0113] As shown in FIG2 , the storage system provided in this embodiment includes a computing node cluster and a storage node cluster. The computing node cluster includes one or more computing nodes 110 (three computing nodes 110 are shown in FIG2 , but are not limited to three computing nodes 110), and each computing node 110 can communicate with each other. The computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, as shown in FIG2 , the computing node 110 includes at least a processor 112, a memory 113, and a network card 114. Among them, the processor 112 is a central processing unit (CPU) that is used to process data access requests from outside the computing node 110 or requests generated within the computing node 110. Exemplarily, when the processor 112 receives write requests sent by a user, it temporarily stores the data in these write requests in the memory 113. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, processor 112 is used to perform data calculations or processing, such as metadata management, deduplication, data compression, virtualized storage space, and address translation. Figure 2 shows only one processor 112. In actual applications, there are often multiple processors 112, each of which has one or more processor cores. This embodiment does not limit the number of processors or processor cores.

[0114] Memory 113 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a very high speed. It serves as a temporary data storage for the operating system or other running programs. The memory includes at least two types of memories. For example, the memory can be either a random access memory RAM or a read-only memory ROM. In actual applications, multiple memories 113 and different types of memories 113 can be configured in the computing node 110. This embodiment does not limit the number and type of memory 113. In addition, the memory 113 can be configured to have a power-saving function. The power-saving function means that when the system loses power and then powers on again, the data stored in the memory 113 will not be lost. A memory with a power-saving function is called a non-volatile memory.

[0115] The network card 114 is used to communicate with the storage node 100. For example, when the total amount of data in the memory 113 reaches a certain threshold, the computing node 110 can send a request to the storage node 100 through the network card 114 to persistently store the data. In addition, the computing node 110 can also include a bus for communication between the components inside the computing node 110. Functionally, since the main function of the computing node 110 in Figure 1 is computing business, remote storage can be used to achieve persistent storage when storing data, so it has less local storage than a conventional server, thereby saving cost and space. However, this does not mean that the computing node 110 cannot have local storage. In actual implementation, the computing node 110 can also have a small number of hard disks built in, or a small number of external hard disks.

[0116] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 (three storage nodes 100 are shown in Figure 1, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other type of storage medium, such as a solid-state drive or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0117] It should be noted that when the network card 114 or the network card 104 is a smart network card, the functions of the processor 112, such as deduplication, can also be offloaded to the smart network card. A smart network card refers to a network card that integrates computing resources, such as a network card with a data processing unit (DPU). The DPU has the versatility and programmability of a CPU, but is more specialized and can run efficiently on network data packets, storage requests, or analysis requests. By offloading functions such as deduplication to the DPU, on the one hand, the use of CPU resources can be reduced, and on the other hand, the access path can be shortened.

[0118] The storage system shown in Figure 2 is a distributed storage system with separated storage and computing. In some possible implementations, the storage system can also be a distributed storage system with integrated storage and computing. The distributed storage system with integrated storage includes a storage cluster (also called a storage node cluster). The storage node cluster can include one or more servers, and the servers can communicate with each other. A server is a device that has both computing and storage capabilities. In terms of hardware, a server includes at least a processor, memory, a network card, and a hard disk. The processor is used to process data access requests from outside the server (application server or other server), and is also used to process requests generated within the server. For example, when the processor receives a write request, it temporarily stores the data in the write request in the memory. When the total amount of data in the memory reaches a certain threshold, the processor sends the data stored in the memory to the hard disk for persistent storage. The processor is also used to calculate or process data, such as metadata management, deduplication, data compression, data verification, virtualized storage space, and address conversion.

[0119] It should be noted that the above is only an example description of the storage system. In other possible implementations of the embodiments of the present application, the storage system may also be a distributed storage system with a fully converged architecture or a distributed storage system with a memory fabric architecture. Detailed description is omitted here.

[0120] In order to implement the deduplication function, the storage system usually needs to introduce "two-level metadata mapping".

[0121] In general, for example, in a storage system that does not support deduplication, metadata mapping can be a direct mapping of a logical block address (LBA) to a physical block address (PBA). The logical block address is also called a logical address, and the physical block address is also called a physical address. The logical address is the address of the logical space presented by the storage medium to the host. When the host sends a write request or a read request to the storage medium, it carries the logical address in the write request or the read request. When the storage medium receives a write request or a read request, it obtains the logical address carried by the write request or the read request, determines the physical address through one or more address conversions of the logical address, and writes data to the physical address or reads data from the physical address. By using LBA as the address of the data, the three-dimensional addressing method of the physical address based on the head, cylinder, and sector is converted into a one-dimensional linear address, which can improve the efficiency of addressing.

[0122] The mapping from logical address to physical address is a single-hop mapping. After the deduplication function is introduced, since the logical address and data content do not match, it is difficult to find the data if routing according to the logical address. Therefore, it is necessary to add a hop of fingerprint routing. Therefore, the single-hop mapping can be changed to a two-level mapping from logical address to fingerprint, and then from fingerprint to physical address.

[0123] In the storage system shown in Figure 1 or Figure 2, the processor (for example, the processor 123 in Figure 1 or the processor 112 in Figure 2) can load computer-readable instructions corresponding to the data deduplication method into the memory (for example, the memory 124 in Figure 1 or the memory 113 in Figure 2), and then the processor executes the above computer-readable instructions to execute the data deduplication method, thereby saving storage space and improving storage performance.

[0124] For ease of description, an example is given in which a storage system performs post-deduplication on data generated by an application.

[0125] Specifically, when an application writes data, the storage system (e.g., a processor within the storage system) can partition the data into blocks. In this example, the data is partitioned into multiple blocks at a 4-kilobyte (KB) granularity. The deduplication module within the storage system can then calculate a fingerprint for each 4KB block, write the fingerprint to a log file, and then save the block to disk (a storage device such as a hard disk). Users can manually trigger deduplication operations or set a deduplication cycle. In this way, the application can issue a deduplication command in response to the user's deduplication operation, or periodically issue a deduplication command to the deduplication module. In response to the deduplication command, the deduplication module sorts the fingerprints in the log file, merges the sorted fingerprints with the fingerprints in the fingerprint file, and deletes data blocks with duplicate content based on the merged result.

[0126] As shown in Figure 3, for data blocks newly written to storage media such as hard disks (which can also be storage devices), the metadata such as the fingerprint of the data block can be managed by appending it to a log file (Write-ahead Log, WAL). When deduplication is triggered, the deduplication module sorts the fingerprints in the log file and merges the sorted fingerprints with the fingerprints in the fingerprint file. Based on the merge result, one data block from multiple data blocks with the same fingerprint can be retained, data blocks with duplicate content can be deleted, and the fingerprint file can be updated. It should be noted that the rectangular blocks with different patterns in Figure 3 represent different fingerprints.

[0127] However, many applications often have uneven workloads. This means that the data written to storage devices such as hard drives for persistent storage can be updated at varying frequencies. Frequently updated data typically accounts for a higher percentage than infrequently updated data. Frequently updated data is often difficult to deduplicate, meaning that the proportion of data that is difficult to deduplicate is higher. This can lead to resource congestion, while infrequently updated data receives a lower percentage of allocated space, making it more susceptible to eviction and resulting in a loss of deduplication efficiency.

[0128] In view of this, the data deduplication method provided in the present application can actively partition the metadata management structure. After the data block in the write request is written to a storage device (such as a hard disk), the metadata such as the fingerprint and address information of the data block is written to the partition corresponding to the characteristics of the data block in multiple partitions of the metadata management structure. When there is a fingerprint identical to the fingerprint of the data block in the partition, the metadata of the data block in the partition is deleted, and the data block is deleted from the storage device according to the address information of the data block.

[0129] In this way, the metadata of data blocks with different characteristics can be written into different partitions of the metadata management structure. For example, the metadata of data blocks that are updated frequently can be written into a partition with a smaller capacity, and the metadata of data blocks that are updated infrequently can be written into a partition with a larger capacity. This prevents data that is updated infrequently from being squeezed out by data that is updated frequently and then eliminated, thereby improving the deduplication rate.

[0130] Furthermore, the present application introduces a new metadata management structure, namely an inverse mapping table (Inverse Mapping Table) for storing the mapping relationship between the fingerprint of the data block and the logical address. As shown in Figure 4, different from the metadata management based on the log file, the embodiment of the present application manages metadata such as fingerprints and logical addresses through an inverse mapping table. When the number of fingerprints in the partition of the inverse mapping table that are the same as the fingerprint of the currently written data block reaches a preset threshold, deduplication is triggered. There is no need to wait for the user to manually trigger or periodically trigger it. The duplicate fingerprints and other metadata in the memory can be deleted in time, reducing the metadata memory overhead and ensuring system performance. In addition, the logical address and physical address of the currently written data block can be written into the address mapping table, and the physical address of the duplicated data block can be modified to the fingerprint. In this way, the duplicated data block can be addressed through two-level mapping, and the non-deleted data block can be addressed through single-level mapping, which shortens the response time of the non-deleted data block.

[0131] It should be noted that the fingerprints and logical addresses in the deduplicated inverse mapping table can also be written to the fingerprint table. This allows the fingerprint table to store the fingerprints and logical addresses of the deduplicated data blocks. The storage system can also use the fingerprint table and the forward mapping table for addressing. Based on this, each partition of the inverse mapping table can also eliminate the metadata in the partition when the elimination conditions are met, thereby reducing the metadata size and memory overhead.

[0132] In order to make the technical solution of the present application clearer and easier to understand, the data deduplication method provided by the present application will be introduced below by taking the storage system 120 shown in FIG1 as an example.

[0133] 5 , a flow chart of a data deduplication method is shown. A storage system 120 includes an engine 121 and a disk enclosure 130 . The engine 121 includes a controller 0 and a controller 1 that back up each other. For ease of description, FIG5 illustrates the method from the perspective of controller 0 . The disk enclosure 130 includes multiple hard disks 134 . The method includes the following steps:

[0134] S502 : Controller 0 receives a write request from application server 100 .

[0135] A write request is a request for writing data. A write request includes data, and the write request is used to write the data to the hard disk 134 for persistent storage. A write request can be generated by an application deployed on the application server 100 based on business needs. For example, a video application can be deployed on the application server 100. The video application can be a short video application or a long video application. The video application can generate a write request, wherein the write request includes a video stream uploaded by the user. For another example, a file management application can be deployed on the application server 100. The file management application can be a file manager. The file manager can generate a write request, wherein the write request includes an image to be archived.

[0136] In the example of FIG. 1 , the controller 0 includes a processor 123 and a front-end interface 125 . The processor 123 of the controller 0 can receive a write request forwarded by the application server 100 through the switch 110 through the front-end interface 125 .

[0137] S504: Controller 0 divides the data in the write request into blocks to obtain at least one data block.

[0138] In this embodiment, the controller 0 can use fixed-length chunking or variable-length chunking to chunk the data in the write request, thereby obtaining at least one data chunk. Fixed-length chunking refers to chunking the data stream according to a set chunking granularity. Variable-length chunking divides the data stream into data chunks of varying sizes. Variable-length chunking can include sliding-window-based variable-length chunking and content-defined chunking (CDC).

[0139] For ease of understanding, the following example is explained using fixed-length blocks. Specifically, when the size of the data stream is an integer multiple of the block granularity, the controller 0 can evenly divide the data into one or more data blocks. When the data size is not an integer multiple of the block granularity, the controller 0 can pad the data, for example, by padding the end of the data with zeros so that the padded data is an integer multiple of the block granularity, and then the controller 0 evenly divides the data into one or more data blocks according to the block granularity. For example, when the size of the data is 19KB, the controller 0 can pad the end of the data with zeros so that the size of the padded data is 20KB, and then the controller 0 blocks the data according to the 4KB block granularity, and 5 data blocks of 4KB in size can be obtained.

[0140] Considering the varying input / output (IO) patterns, IO sizes, and feature requirements across different storage scenarios, Controller 0 can select an appropriate block partitioning strategy based on the storage scenario. For example, in primary storage scenarios, where IO is typically small and the IO pattern is primarily random read / write, Controller 0 can choose fixed-length block partitioning. In backup storage scenarios, where IO is typically large and the IO pattern is primarily sequential read / write, Controller 0 can choose variable-length block partitioning to achieve a better deduplication rate.

[0141] It should be noted that data segmentation is an optional step in the data deduplication method of the embodiments of the present application, and the data deduplication method of the embodiments of the present application does not necessarily need to perform the above steps. For example, when the data size is equal to or smaller than the segmentation granularity, the data can be directly treated as a data block. For another example, when the data size is fixed, the data can also be directly treated as a data block.

[0142] S506: Controller 0 determines a fingerprint of at least one data block.

[0143] For any data block in the at least one data block, the controller 0 may calculate the fingerprint of the data block using a message digest algorithm based on the content of the data block. The fingerprint of the data block may be a message digest of the data block, such as a hash value of the data block.

[0144] S508: Controller 0 queries the fingerprint table based on the fingerprint of the first data block in the at least one data block. If the fingerprint of the first data block exists in the fingerprint table, S509 is executed. If the fingerprint of the first data block does not exist in the fingerprint table, S510, S511, and S512 are executed.

[0145] S509 : The controller 0 returns a write response to the application server 100 .

[0146] S510 : Controller 0 writes the first data block to hard disk 134 .

[0147] The fingerprint table is used to record the fingerprint and address information of the data blocks stored on the hard disk 134. The address information may include a logical address. Furthermore, the address information may also include a physical address. The fingerprint table may use a key-value (kv) pair to store fingerprints and address information. Specifically, the fingerprint table may use the fingerprint as the key and address information such as the logical address as the value.

[0148] The fingerprint and address information recorded in the fingerprint table can come from the inverse mapping table. The inverse mapping table is a metadata management structure that stores the fingerprints and logical addresses of data blocks that have been downloaded to disk. Specifically, after the inverse mapping table triggers deduplication, controller 0 can synchronize the deduplicated inverse mapping table with the fingerprint table. Specifically, the metadata (e.g., fingerprints and logical addresses) in the deduplicated inverse mapping table can be stored in the fingerprint table.

[0149] The fingerprint table stores the fingerprints of data blocks that have been downloaded to disk, thereby supporting pre-deduplication and reducing the storage pressure on the hard disk 134. Specifically, the controller 0 can query the fingerprint table based on the fingerprint of the first data block. For example, the controller 0 can compare the fingerprint of the first data block with the fingerprints in the fingerprint table, or the controller 0 can quickly search for the fingerprint based on the fingerprint of the first data block and the index of the fingerprint table.

[0150] If the fingerprint of the first data block exists in the fingerprint table, indicating that a data block with the same content has been written to hard disk 134, controller 0 can execute S509 to directly return a write response, which indicates that the write was successful. If the fingerprint of the first data block does not exist in the fingerprint table, indicating that a data block with the same content is not stored on disk 134, controller 0 can execute S510 to write the first data block to disk 134. Furthermore, controller 0 can also return a write response to application server 100 after the first data block is successfully written to hard disk 134.

[0151] It should be noted that in the initial stage of the service, the fingerprint table can be empty. As the application server 100 continuously stores data to the hard disk 134, the metadata of the data blocks that have been downloaded can be recorded in the inverse mapping table. When the partition in the inverse mapping table triggers deduplication, the metadata in the deduplicated inverse mapping table can be synchronized to the fingerprint table. At this stage, the controller 0 can query the fingerprint table to achieve front-end deduplication. Compared with post-deduplication, front-end deduplication eliminates the need to write duplicate data to storage media such as the hard disk 134 by deleting duplicate data before the data is downloaded to the disk, thereby avoiding resource occupation.

[0152] Based on this, the method of the embodiment of the present application can also be executed without executing S508 and S509. For example, at the initial node of the service, the fingerprint table is empty, and controller 0 can directly save the data block to disk for post-deduplication. When the metadata recorded in the fingerprint table reaches a preset number of entries, front-end deduplication is supported. For another example, controller 0 can directly save the data block to disk for post-deduplication without performing front-end deduplication.

[0153] S511: Controller 0 writes the logical address and physical address of the first data block into the address mapping table.

[0154] The address mapping table is used to store address information for data blocks written to hard disk 134. For example, the address mapping table can store the logical address and physical address of a data block. In a specific implementation, the address mapping table can use a key-value (KV) format to store logical and physical addresses. To facilitate searching or locating data blocks, the address mapping table can use the logical address as the key and the physical address as the value to store the mapping relationship between the logical address and the physical address. This allows for quick addressing when accessing the data block later, and also allows for recording the operation to facilitate subsequent tracing or fault recovery.

[0155] Different from the reverse mapping table which uses the logical address as the value, the address mapping table uses the logical address as the key, so it can also be called a forward mapping table. It should be noted that if the first data block is forward-deduplicated, it means that the first data block has not been written to the hard disk 134, and there is no corresponding physical address. The controller 0 can store the logical address and fingerprint of the first data block in the forward mapping table. When it is necessary to find the first data block, the fingerprint of the first data block can be found according to the forward mapping table, and then the fingerprint table can be found to obtain the data block with the same fingerprint as the first data block, and the logical address of the data block with the same fingerprint can be obtained. Based on the logical address of the data block with the same fingerprint, the physical address of the data block with the same fingerprint can be obtained through the forward mapping table. Based on the physical address of the data block with the same fingerprint, the data block with the same fingerprint can be accessed, thereby achieving access to the first data block.

[0156] It should be noted that the execution order of the above S510 and S511 is not limited. In some possible implementations, the controller 0 may first write the logical address and the physical address into the forward mapping table, and then write the first data block into a storage device such as the hard disk 134.

[0157] S512: Controller 0 inserts the fingerprint and logical address of the first data block into the first partition of the inverse mapping table. If there is a fingerprint identical to the fingerprint of the first data block in the first partition, S514 and S516 are executed.

[0158] The inverse mapping table is used to store fingerprints and logical addresses. The inverse mapping table can be a table structure organized in the form of key-value pairs. The key-value pairs in the inverse mapping table are used to represent the inverse mapping relationship. Different from the mapping form from logical address to physical address in the address mapping table, the inverse mapping table is used to store the inverse mapping relationship from fingerprint to logical address. The key in the key-value pair is the fingerprint, and the value is the logical address. Controller 0 can insert the fingerprint and logical address of the first data block into the inverse mapping table in an orderly manner.

[0159] The controller 0 may perform N-Way Merge sorting on the fingerprint of the first data block and the fingerprint of the first partition in the inverse mapping table, and then insert the fingerprint and the logical address of the first data block into the first partition of the inverse mapping table according to the sorting result.

[0160] Multi-way merge sorting means that the objects to be sorted (for example, fingerprints of data blocks) are divided into multiple ways and sorted separately, and then the sorting results of each way are merged to achieve merge sorting. Assume that the sorting result of each way can be recorded as an ordered small set S with n elements = {x|x i ≤x j ,i,j∈[0,n)}, assuming that the fingerprints to be sorted are divided into m paths, then m small sets S1, S2, ...S m Merge to achieve merge sort. Controller 0 can determine the minimum value in all small sets according to the following formula and write it into the large set: min=min(min(S1),min(S2),…min(S m )) (1)

[0161] Assume that the minimum value among all small sets is set S i The controller 0 can write the minimum value into the large set and then remove the minimum value from the set S. i Remove, and then continue to calculate according to the above formula (1) to determine the minimum value of all updated small sets, and write the minimum value into the large set. Controller 0 repeats the above process until all elements in the small sets are written into the large set, completing the merge sort.

[0162] In some possible implementations, the controller 0 may use a log structured merge tree (LSM tree) to sort the fingerprint of the first data block and the fingerprints in the inverse mapping table, and then sequentially insert the fingerprint and logical address of the first data block into the first partition of the inverse mapping table. Each partition of the inverse mapping table may maintain an LSM tree, thereby sequentially inserting metadata such as the fingerprint and logical address to be written to the partition.

[0163] In this embodiment, the inverse mapping table includes multiple partitions, and the first partition can be determined based on the characteristics of the first data block. The characteristics of the first data block can be the fingerprint corresponding to the first data block. In some cases, multiple data blocks can correspond to the same fingerprint. For example, in a backup scenario, multiple data blocks can correspond to the same fingerprint. Controller 0 can determine the heat of the fingerprint corresponding to the first data block based on the heat of the fingerprint, and then determine the first partition among the multiple partitions of the inverse mapping table corresponding to the heat based on the heat of the fingerprint corresponding to the first data block. For example, controller 0 can compare the heat of the fingerprint with a preset heat. When the heat of the fingerprint is less than the preset heat, controller 0 can determine that the first partition is a partition in the inverse mapping table for storing metadata for cold data, also known as a cold partition. When the heat of the fingerprint is greater than or equal to the preset heat, controller 0 can determine that the first partition is a partition for storing hot data, also known as a hot partition. The cold partition can be a partition with a larger capacity, and the hot partition can be a partition with a smaller capacity. The capacity of the cold partition is greater than that of the hot partition. Controller 0 then writes the fingerprint and logical address of the first data block into the first partition.

[0164] Furthermore, after writing metadata such as the fingerprint and logical address of the first data block to the first partition, controller 0 may also update the heat of the fingerprint corresponding to the first data block to determine the partition corresponding to a subsequently written data block having the same fingerprint. Specifically, controller 0 may determine the heat of the logical address of the first data block, add the heat of the logical address to the heat of the fingerprint corresponding to the first data block, and thereby update the heat of the fingerprint corresponding to the first data block.

[0165] For ease of understanding, the following is an example. In this example, the controller 0 can update the heat of the fingerprint corresponding to the data block after each metadata of the data block is written. For example, the first data block that the controller 0 writes to the hard disk 134 is data block 10, and the fingerprint of data block 10 is recorded as FP3. Since a data block with the same fingerprint has been written before, and the data block with the fingerprint FP3 written most recently is data block 8, the controller 0 can obtain the heat of FP3 updated after the metadata of data block 8 is written. In this example, it is assumed that the heat of FP3 is 5. Based on the heat, the controller 0 can determine the first partition of multiple partitions of the inverse mapping table. For example, the first partition can be a partition for storing metadata of cold data, also called a cold partition. The controller 0 can also determine the heat of the logical address of data block 10. Assuming that the heat of the logical address is 2, the heat of the logical address can be accumulated to the heat of the fingerprint, thereby updating the heat of the fingerprint. In this example, the heat of the updated fingerprint can be 7.

[0166] As data blocks are continuously written, the heat of the fingerprint corresponding to the data block may change. To this end, the controller 0 can also move the metadata of the data block in different partitions according to the heat of the fingerprint corresponding to the data block. For example, in the initial stage, the heat of the fingerprint corresponding to each data block is usually low, and the metadata of these data blocks can be written to the cold partition. As data blocks are continuously written, the heat of some fingerprints continues to increase. When the heat of the fingerprint corresponding to a data block is greater than the preset heat after writing a data block, the controller 0 can write the metadata of the data block to the hot partition, and move the metadata of the data block with the same fingerprint to the hot partition.

[0167] In some possible implementations, considering the overhead of moving metadata between different partitions, controller 0 may not move the metadata but instead eliminate it. For example, after writing a data block, if the popularity of the fingerprint corresponding to the data block is greater than a preset popularity, controller 0 may write the metadata of the data block to the hot partition and eliminate the metadata of the data block with the same fingerprint from the cold partition.

[0168] Furthermore, the controller can trigger deduplication when a fingerprint identical to the fingerprint of the first data block exists in the first partition, or trigger deduplication when a fingerprint identical to the fingerprint of the first data block exists in the first partition and the number of fingerprints identical to the fingerprint of the first data block in the first partition reaches a preset threshold, and execute S514 and S516.

[0169] Among them, the preset threshold value can be set according to an empirical value. For example, the preset threshold value can be set to 2. Taking Figure 4 as an example, in this example, if the number of fingerprints identical to the fingerprint of the first data block in partition 1 (such as the first partition) of the inverse mapping table reaches 2, deduplication can be triggered. For another example, the preset threshold value can also be set to 1, that is, the existence of a fingerprint identical to the fingerprint of the first data block can trigger deduplication. That is to say, when there is a fingerprint identical to the fingerprint of the first data block in the first partition, the controller 0 can execute S514 and S516 to implement deduplication, but the embodiment of the present application does not limit the timing of deduplication. For example, the controller 0 can trigger deduplication when the number of identical fingerprints in the first partition reaches the preset threshold, or it can trigger deduplication when the identical fingerprint exists.

[0170] In this embodiment, the controller 0 directly writes the first data block to the hard disk 134 and then performs post-deduplication, which can avoid wasting computing resources and affecting normal business operations, and avoid storage performance degradation caused by computing bottlenecks.

[0171] It should be noted that the inverse mapping table is a new metadata management structure introduced in this embodiment. In other possible implementations of the embodiment of the present application, the metadata management structure may also adopt other organizational forms.

[0172] S514 : The controller 0 deletes the first data block from the hard disk 134 according to the address information of the first data block.

[0173] For data blocks with the same fingerprint, the controller 0 may retain one data block and delete the other data blocks with the same fingerprint from the hard disk 134. In a specific implementation, the controller 0 may obtain the physical address of the first data block from the address mapping table based on the logical address of the first data block, then find the first data block from a storage device such as the hard disk 134 based on the physical address, and delete the first data block.

[0174] Furthermore, when the preset threshold is greater than 1, the controller 0 may retain a data block with the same fingerprint as the first data block and delete other data blocks with the same fingerprint. Specifically, for data blocks with the same fingerprint, the controller 0 may retain the data block written first and delete the data blocks written later.

[0175] S516: Controller 0 deletes the fingerprint and logical address of the first data block from the first partition of the inverse mapping table.

[0176] Specifically, the controller 0 can search for the fingerprint of the first data block from the inverse mapping table, and then delete the fingerprint and logical address of the first data block. It should be noted that when the key-value pairs of fingerprints and logical addresses in the inverse mapping table are stored in the form of LSM tree, the controller 0 can delete the fingerprint of the first data block and the corresponding logical address by table merging.

[0177] Furthermore, when the preset threshold is greater than 1, controller 0 may retain metadata for a data block with the same fingerprint as the first data block and delete metadata for other data blocks with the same fingerprint. Specifically, for data blocks with the same fingerprint, controller 0 may retain metadata for the data block written first and delete metadata for data blocks written later.

[0178] In addition, the controller 0 may execute S514 and S516 in parallel or in sequence. For example, the controller 0 may execute S516 first and then S514. This embodiment does not limit the execution order of S514 and S516.

[0179] It should be noted that at least one data block may also include a second data block. The heat of the fingerprint corresponding to the second data block may be higher than the heat of the fingerprint corresponding to the first data block. For example, the second data block may be hot data, and the first data block may be cold data. Accordingly, the controller 0 may write the second data block to the hard disk 134, and write the metadata of the second data block, such as the fingerprint and logical address of the second data block, to the second partition among the multiple partitions of the inverse mapping table. The second partition is specifically a hot partition for storing metadata of hot data. The capacity of the second partition is smaller than that of the first partition.

[0180] Furthermore, the heat of the fingerprint corresponding to the data block can be divided into more types or levels. For example, the heat of the fingerprint can be divided into three levels: hot, warm, and cold. Accordingly, the inverse mapping table can include more partitions. For example, the inverse mapping table can include a partition for storing metadata for hot data, a partition for storing metadata for warm data, and a partition for storing metadata for cold data.

[0181] The above S516 is an implementation method for deleting the metadata of the first data block in the first partition in an embodiment of the present application. When the metadata management structure is used to store fingerprints, logical addresses and physical addresses, controller 0 can delete the fingerprints, logical addresses and physical addresses of the first data block in the first partition of the metadata management structure.

[0182] S518: Controller 0 writes the fingerprint and logical address in the deduplicated inverse mapping table into the fingerprint table.

[0183] Specifically, the controller 0 can also write metadata such as fingerprints and logical addresses in the deduplicated inverse mapping table to the fingerprint table in a synchronous manner. Among them, the controller 0 can synchronously write the metadata in the deduplicated inverse mapping table into the fingerprint table with partitions as the granularity. For example, after the first partition triggers deduplication, the controller 0 can synchronously write the metadata of the first partition after deduplication into the fingerprint table. After the second partition triggers deduplication, the controller 0 then synchronously writes the metadata of the second partition after deduplication into the fingerprint table. Considering that a partition can trigger multiple deduplications, in order to reduce resource usage, the controller 0 can adopt an incremental synchronization mechanism to synchronously write the metadata after deduplication into the fingerprint table.

[0184] S520: When at least one partition in the inverse mapping table meets the elimination condition, the controller 0 eliminates metadata in the at least one partition.

[0185] Specifically, the partitions in the inverse mapping table can be set with a water level for elimination. When the amount of resources occupied by the metadata in the partition reaches the water level, the controller 0 can eliminate the metadata in the partition, thereby avoiding metadata overflow. Since different partitions store fingerprints of different popularity, the water levels of the partitions can be different. For example, when the capacity of the first partition is 80% of the total capacity of the inverse mapping table and the capacity of the second partition is 20% of the total capacity of the inverse mapping table, the water level of the first partition can be 70% of the total capacity of the inverse mapping table, and the water level of the first partition can be 10% of the total capacity of the inverse mapping table.

[0186] In some possible implementations, the watermarks for partitions in the inverse mapping table may include a high watermark and a low watermark. When the amount of resources occupied by metadata in a partition reaches the high watermark, the metadata in the partition may be eliminated so that the amount of resources occupied by metadata in the partition after elimination is no less than the low watermark and no more than the high watermark.

[0187] Controller 0 eliminates metadata in the partitions of the inverse mapping table, especially metadata in the partitions corresponding to frequently updated hot data, which can significantly reduce the metadata size and reduce memory overhead.

[0188] S522: Controller 0 modifies the physical address of the first data block to be deleted in the address mapping table into the fingerprint of the first data block.

[0189] Since the first data block has been deleted from the hard disk 134, it is difficult to locate the first data block based on the physical address in the address mapping table. Therefore, the controller 0 can modify the physical address of the deleted first data block in the address mapping table to the fingerprint of the first data block, indicating that the first data block is a deleted data block. The data block with the same fingerprint as the data block can be found through the fingerprint table, and then the physical address of the data block with the same fingerprint can be determined, so that the addressing of the deleted first data block can be achieved.

[0190] Based on this, the undeleted data blocks can still be addressed based on the mapping relationship between the logical address and the physical address in the forward mapping table, without the need for two-level metadata mapping, which greatly shortens the addressing time and thus shortens the response time and improves the response efficiency.

[0191] It should be noted that the above S518 to S522 are optional steps of the embodiment of the present application. For example, when the memory is a large-capacity memory, the controller 0 may not execute the steps of writing the data in the inverse mapping table into the fingerprint table and eliminating the metadata in the inverse mapping table.

[0192] Based on the above description, the data deduplication method provided by this application actively partitions the metadata management structure to differentially process the metadata of data with different update frequencies. Among them, data that is not updated frequently can be allocated sufficient quota resources to support deduplication, while data that is updated frequently is frequently invalidated, so the allocated quota resources are relatively small; this ensures that the data that can be deduplicated can basically be deduplicated instead of being squeezed out and eliminated, thereby improving the deduplication rate; at the same time, data that is difficult to dedupe can be eliminated, thereby reducing the overall metadata mapping scale and improving system performance.

[0193] Furthermore, the method introduces a novel metadata management structure called an inverse mapping table. When a fingerprint identical to the fingerprint of the first data block exists in the first partition of the inverse mapping table, deduplication can be triggered without the need for manual deduplication by the user. Alternatively, deduplication can be triggered periodically, enabling timely deduplication and reducing metadata size, thereby reducing metadata memory overhead and ensuring system performance. Furthermore, the method records the logical and physical addresses of the disk's data blocks in the inverse mapping table and modifies the physical addresses of the deduplicated data blocks to fingerprints. This allows non-deduplicated data blocks to still be addressed via single-hop mapping, shortening addressing time and improving response efficiency.

[0194] The key to the embodiment shown in FIG5 lies in partitioning metadata management structures, such as inverse mapping tables. The capacity of each partition in the metadata management structure can be determined based on a partition decision model. The partition decision model is used to predict the corresponding partition benefit of each of the preset partition capacity combinations after application to the metadata management structure, and to determine the partition capacity combination with the greatest partition benefit as the capacity of the multiple partitions in the metadata management structure.

[0195] The partition capacity combination represents the capacity of each partition in a group. Partition capacity refers to the amount of resources allocated to a partition. The sum of the capacities of each partition in a group equals the total capacity of the metadata management structure. Therefore, the partition capacity combination can be represented by the actual capacity of each partition or by the capacity percentage of each partition.

[0196] For ease of description, the following examples use partition capacity combinations and partition capacity ratios. For example, a partition capacity combination can be expressed as 80%:20%, which indicates that the metadata management structure includes two partitions with capacities of 80% and 20% of the total capacity, respectively. Another example is a partition capacity combination expressed as 60%:30%:10%, which indicates that the metadata management structure includes three partitions with capacities of 60%, 30%, and 10% of the total capacity, respectively.

[0197] Partitioning benefits refer to the benefits gained from partitioning the metadata management structure. For example, partitioning the metadata management structure can improve the deduplication rate, so the partitioning benefit can be the deduplication rate.

[0198] The partition decision model can predict the deduplication rate by estimating the data hit rate. For ease of description, the first partition capacity combination among the preset partition capacity combinations is used as an example. The partition decision model can predict the corresponding deduplication rate after the first partition capacity combination is applied to the metadata management structure by the following method:

[0199] Obtain workload characteristics corresponding to each of the multiple partitions formed by applying the first partition capacity combination to the metadata management structure, then obtain data distribution corresponding to each partition based on the workload characteristics corresponding to each partition, and then obtain deduplication rate based on the data distribution corresponding to each partition and the capacity of each partition.

[0200] Furthermore, considering that the workload may change, the capacity of the partition can also be adjusted. The workload can be tasks that are using or waiting to use computing resources such as the CPU over a period of time. During business peak periods, there are more tasks that are using or waiting to use computing resources such as the CPU, and the workload is heavier. During business slack periods, there are fewer tasks that are using or waiting to use computing resources such as the CPU, and the workload is lighter. Based on this, controller 0 can adjust the partition. During the specific project implementation process, partition adjustment requires operations such as re-initialization of the partition, which incurs partition adjustment costs. Based on this, the partition benefit can be determined based on at least one of the rate of return and the partition adjustment cost.

[0201] In some possible implementations, the controller 0 may periodically adjust the capacity of multiple partitions of the metadata management structure. Each period may also be referred to as a partition adjustment period. When the adjustment moment is reached, the controller 0 may determine whether to adjust the capacity of multiple partitions based on the partition revenue corresponding to the period before the adjustment moment (e.g., the previous partition adjustment period), the partition capacity combination, or the workload characteristics corresponding to each partition. Workload characteristics refer to characteristics extracted from workload information. For example, workload characteristics may include one or more of reuse distance, reuse period, and reuse frequency.

[0202] For ease of understanding, the partitioning process in the data deduplication method of the present application is described in detail below with reference to the accompanying drawings. For ease of description, the partitioning of the inverse mapping table is still used as an example for description.

[0203] Referring to the flowchart of the data deduplication method shown in FIG6 , based on the embodiment shown in FIG5 , the controller 0 may further perform the following steps after writing the first data block to the hard disk 134:

[0204] S602: Controller 0 obtains system resource usage information of the last partition adjustment cycle and workload information corresponding to each partition.

[0205] System resources include one or more of computing resources (such as processor resources such as CPU), memory resources, disk resources, and network resources. Based on this, system resource usage information includes CPU usage ratio, memory usage ratio, disk IO volume, and bandwidth usage.

[0206] Workload refers to a task that is using or waiting to use computing resources such as the CPU within a period of time. The task may be writing data. Based on this, the workload information may include one or more of the data's reuse distance, reuse cycle, or reuse frequency. The reuse distance may be the number of accesses between two adjacent accesses to the same data. When counting the reuse distance, statistics may be performed at the granularity of data blocks. The reuse cycle may be the number of write requests between two adjacent accesses to the same data in different write requests. The reuse frequency may be the inverse of the reuse cycle. It should be noted that when the controller 0 obtains workload information such as the reuse distance, reuse cycle, or reuse frequency, it may extract the reuse distance, reuse cycle, or reuse frequency for each partition separately, thereby obtaining the workload information corresponding to each partition.

[0207] S604: Controller 0 extracts system resource features from the system resource usage information, and extracts workload features corresponding to each partition from the workload information corresponding to each partition.

[0208] Specifically, the controller 0 may vectorize the system resource usage information to obtain the system resource characteristics, and vectorize the workload information to obtain the workload characteristics. For ease of understanding, the vectorization of CPU occupancy ratio and reuse distance is used as an example for explanation.

[0209] When vectoring the CPU usage ratio, controller 0 may compare the CPU usage ratio with a CPU usage threshold. The CPU usage threshold may be set based on historical business experience. For example, the CPU usage threshold may be set to 70%. When the CPU usage ratio is higher than the CPU usage threshold, a "1" may be output; when the CPU usage ratio is lower than the CPU usage threshold, a "0" may be output. For details, see the following formula:

[0210] Among them, F represents the feature, and F can be represented by a vector.

[0211] Similarly, referring to the schematic diagram of system resource feature extraction shown in Figure 7, controller 0 can compare system resource usage information such as memory occupancy ratio, disk IO amount, bandwidth occupancy, etc. with the threshold value corresponding to the resource, and output the system resource features corresponding to the above system resource usage information based on the comparison results, such as F (memory occupancy), F (disk IO amount), and F (bandwidth occupancy).

[0212] When vectorizing the reuse distance, the reuse distance of the current batch of data (such as the data blocks included in the current write request) can be accumulated, and then the mean and variance of the reuse distance can be calculated based on this, as shown below:

[0213] Similarly, controller 0 can use the same statistical method as that used to process the reuse distance to perform statistics on the reuse period and reuse cycle to fully explore the temporal correlation and spatial correlation therein, thereby extracting workload features from the workload information.

[0214] It should be noted that the above S602 and S604 are optional implementations of the embodiment of the present application, and the method of the embodiment of the present application may not perform the above S602 and S604. For example, the controller 0 may not obtain system resource usage information and extract system resource features.

[0215] S606: Controller 0 obtains structural features according to system resource features and workload features.

[0216] Structured features include system resource features and workload features. The system resource features are extracted from system resource usage information, while the workload features are extracted from workload information. Controller 0 can fuse the system resource features and workload features to generate structured features. For example, Controller 0 can concatenate system resource features and workload features to generate structured features.

[0217] Among them, controller 0 can also merge system resource features. From a business perspective, if a certain system resource is occupied too much at certain times, it will cause a system performance bottleneck. At this time, the same type of system resource features (also called associated features, common impact features) are no longer necessary. For such features, feature merging can be performed. As shown in Figure 8, controller 0 can use "or" operation to achieve feature merging. For example, F (CPU occupancy), F (memory occupancy), F (disk IO volume), and F (bandwidth occupancy) are common impact features, and controller 0 can merge the above common impact features.

[0218] As shown in Figure 9, controller 0 obtains workload characteristics and system resource characteristics with common influences through multi-source information processing. After merging the common influence characteristics, controller 0 can standardize and normalize the common influence characteristics. Similarly, controller 0 can standardize and normalize the workload characteristics after extracting the workload characteristics. Among them, the standardized and normalized system resource characteristics can be a0a1…a k , the workload characteristics after standardization and normalization can be b0b1…b k Controller 0 can use the above system resource characteristics a0a1…a k and workload characteristics b0b1…b k Splicing is performed to obtain structural features.

[0219] Controller 0 cleans the features through common feature processing methods such as feature merging and normalization, which can generate a feature model that accurately describes the current situation with low computational overhead, providing a reliable basis for partitioning decisions.

[0220] S602 to S606 above are one implementation of controller 0 acquiring structured features. In other possible implementations of the present embodiment, controller 0 may also acquire structured features through other methods. Furthermore, when controller 0 does not acquire system resource usage information and extract system resource features from it, controller 0 may not execute S606 above.

[0221] S608: Controller 0 obtains feedback information of the last partition adjustment cycle and determines whether to trigger partition adjustment based on the feedback information of the last partition adjustment cycle. If yes, execute S610; if not, execute S622.

[0222] Controller 0 can set the trigger conditions for partition adjustment. Controller 0 can determine whether the trigger conditions for partition adjustment are met based on feedback from the previous partition adjustment cycle, such as the partition benefits (such as deduplication rate) and the workload characteristics of each partition, and thus determine whether to trigger the partition adjustment.

[0223] For the current partition adjustment cycle, the trigger conditions for partition adjustment can be set as follows: the deduplication rate in the previous partition adjustment cycle is less than a preset value or the decrease reaches a preset range, or the workload characteristics of the previous partition adjustment cycle compared to the workload characteristics of the previous partition adjustment cycle meet preset conditions. The preset value, preset range, or preset condition can be set based on historical business experience.

[0224] For example, the workload characteristics of the previous partition adjustment cycle indicate that the workload is mainly large IO, and the workload characteristics of the previous partition adjustment cycle indicate that the workload is mainly small IO. That is, the workload characteristics of the previous partition adjustment cycle have changed significantly compared with the workload characteristics of the previous partition adjustment cycle, which can trigger partition adjustment.

[0225] It should be noted that the data deduplication method of the embodiment of the present application may also be executed without executing S608. For example, the controller 0 may directly trigger the partition adjustment and then perform partition update according to the results of the partition decision modeling.

[0226] S610: The controller 0 determines a target modeling strategy from a set of modeling strategies according to the structural features.

[0227] S612: Controller 0 selects a target evaluation strategy from the partition benefit evaluation strategy set according to the structured features and feedback information.

[0228] S614: Controller 0 determines the objective function of the partition decision model according to the objective evaluation strategy.

[0229] S616: The controller 0 performs partition decision modeling according to the structured features, using the target modeling strategy and the objective function to obtain a partition decision model.

[0230] S618: Controller 0 obtains a partition capacity combination with the maximum partition benefit according to the partition decision model.

[0231] Controller 0 can perform partition decision modeling based on the structured features obtained from multi-source information processing, and complete the partition decision with the assistance of early feedback information, such as the workload characteristics of the previous partition adjustment cycle, the deduplication rate and other prior knowledge. As shown in Figure 10, controller 0 can select a modeling strategy from the modeling strategy set based on the structured features to determine the target modeling strategy, and select an evaluation strategy from the partition benefit evaluation strategy set based on the workload characteristics of the previous partition adjustment cycle, the deduplication rate and other feedback information to determine the target evaluation strategy. The objective function of the partition decision model can be determined based on the target evaluation strategy. Then, controller 0 can perform modeling based on the structured features through the target modeling strategy and the objective function, and then obtain the partition capacity combination with the maximum partition benefit based on the partition decision model obtained by modeling. Among them, the partition capacity combination with the maximum partition benefit can also be called a partition decision.

[0232] In order to make the technical solution of the present application clearer and easier to understand, the following examples illustrate the processes of modeling strategy selection, evaluation strategy selection, partition decision modeling, etc.

[0233] Take the case where the workload is heavy and the system resource usage ratio is large as an example.

[0234] In this example, the modeling strategy set includes: 1. a modeling strategy based on dot-based partitioning; 2. a modeling strategy based on Gaussian process regression. The dot-based partitioning strategy is designed for general, or simple, scenarios, while the Gaussian process regression strategy is designed for complex scenarios. Dot-based partitioning provides multiple preset partition capacity combinations, from which you can select the one that maximizes the partitioning benefit.

[0235] In this example, the workload characteristics in the structured feature vector reflect that the business scenario is a simple scenario. In addition, the flag bit related to the CPU occupancy characteristic is 1, indicating that the CPU and other system resources are occupied more. In order to avoid the partition decision modeling occupying more system resources, controller 0 can choose a modeling strategy based on dot partitioning for modeling.

[0236] In this example, it is assumed that the workload's reuse distance distribution follows a normal distribution, as shown below:

[0237] Where σ represents the variance and μ represents the mean. Controller 0 can use the mean μ of the reuse distance in the structured features obtained by multi-source information processing 重用距离均值 and the variance of the reuse distance σ 重用距离方差 Fit the probability density function of the reuse distance. For details, see the following formula:

[0238] This example uses two partitions for illustration. Specifically, the two partitions are a hot partition corresponding to frequently updated data and a cold partition corresponding to infrequently updated data. Controller 0 can set the following partition combination based on the dot-based partitioning modeling strategy:

[0239] Table 1 Partition combination

[0240] Combined with the reuse distance probability density function of formula (5) above, controller 0 can obtain the hit rate of the two types of data distribution by integration, and then obtain the deduplication rate obtained by this partitioning scheme by multiplying the hit rate and the data proportion, as shown below: f(P) = P*∫F1 + (1-P)*∫F2 (6)

[0241] Among them, F1 and F2 respectively represent the data distribution corresponding to the two partitions, which can be specifically represented by the probability density function of the reuse distance of the data corresponding to the two partitions, and P is the capacity ratio of a partition.

[0242] In this example, controller 0 can select the "maximize deduplication rate" evaluation strategy from the set of partition benefit evaluation strategies based on the structural characteristics as the target evaluation strategy. In other words, controller 0 can directly use the function in formula (6) as the objective function to perform partition decision modeling. Specifically, controller 0 substitutes the parameters of different partition combinations into the objective function to obtain the deduplication rates for each partition combination. Controller 0 then selects the partition capacity combination with the highest deduplication rate.

[0243] The above example is an example of selecting an evaluation strategy directly based on structured features. In some possible implementations, the controller 0 can also reconstruct the evaluation strategy of the partition benefit in combination with other constraints. For example, during the implementation of a specific project, the change in the size of the partitions before and after should not be too large, otherwise it will affect the partition deployment performance and cause unnecessary overhead. Based on this, other constraints may include minimizing the partition adjustment cost. The reconstructed partition benefit evaluation strategy can be an evaluation strategy based on the deduplication rate and the partition adjustment cost.

[0244] As shown in Figure 11, the controller 0 can obtain feedback information of the previous partition adjustment cycle, and the feedback information may include one or more of workload characteristics and deduplication rate. The controller 0 can adjust the evaluation strategy of the partition benefit according to the feedback information. Specifically, if the deduplication rate of the previous partition adjustment cycle is less than the preset value or the decrease reaches a preset range, the controller 0 can adjust the evaluation strategy of the partition benefit to an evaluation strategy based on the deduplication rate and the partition adjustment cost. Similarly, if the change in the workload characteristics of the previous partition adjustment cycle relative to the workload characteristics of the previous partition adjustment cycle meets the preset conditions, the controller 0 can adjust the evaluation strategy of the partition benefit to an evaluation strategy based on the deduplication rate and the partition adjustment cost.

[0245] Compared with the evaluation strategy based on the deduplication rate, the evaluation strategy based on the deduplication rate and partition adjustment cost takes into account multiple factors and is more targeted. The partition capacity combination selected in this way also has a higher partition benefit.

[0246] When the partition benefit evaluation strategy is adjusted to an evaluation strategy based on the deduplication rate and partition adjustment cost, the objective function can be adjusted from the above formula (6) to: f(P) = P*∫F1+(1-P)*∫F2-||PP 当前分区比例 || 2 (7)

[0247] Among them, the above S610 to S616 are an implementation method for controller 0 to construct the partition decision model based on the workload characteristics of the previous partition adjustment cycle. Among them, S618 is an implementation method for determining the partition decision (specifically, the target partition combination). It should be noted that when modeling the partition decision model, controller 0 may not execute the above S612 and S614. For example, controller 0 can perform partition decision modeling based on the target modeling strategy and the default objective function to obtain the partition decision model.

[0248] S620: Controller 0 partitions the inverse mapping table according to the partition capacity combination with the maximum partition benefit.

[0249] The partition capacity combination includes the capacity proportions of different partitions, that is, the resource proportions allocated to different partitions. For example, the partition capacity combination may include the resource proportions allocated to the hot partition and the resource proportions allocated to the cold partition. Controller 0 may partition the system resources in the inverse mapping table based on the resource proportions allocated to the hot partition and the resource proportions allocated to the cold partition.

[0250] For example, the partition capacity combination is that the resource allocated to the hot partition accounts for 20%, and the resource allocated to the cold partition accounts for 80%. Controller 0 can allocate 20% of the storage space in the inverse mapping table to the hot partition and 80% of the storage space in the inverse mapping table to the cold partition.

[0251] S620: Controller 0 writes the fingerprint and logical address of the first data block into the first partition.

[0252] Among them, the first partition is a partition determined from multiple partitions based on the characteristics of the first data block. For example, when the fingerprint corresponding to the first data block has a higher popularity, the first partition can be a hot partition; when the fingerprint corresponding to the first data block has a lower popularity, the first partition can be a cold partition.

[0253] For the specific implementation of the controller 0 writing the fingerprint and the logical address of the first data block into the first partition, reference may be made to the description of the embodiment shown in FIG5 , which will not be repeated here.

[0254] S622: Controller 0 determines whether deduplication is triggered. If so, execute S624; if not, execute S626.

[0255] Referring to the description of the embodiment shown in FIG5 , controller 0 may compare the fingerprint of the first data block with the fingerprint in the first partition of the inverse mapping table. Deduplication may be triggered when a fingerprint identical to the fingerprint of the first data block exists in the first partition. For example, when the number of fingerprints identical to the fingerprint of the first data block in the first partition reaches a preset threshold, deduplication may be performed in step S624 .

[0256] S624: Controller 0 performs deduplication based on the LSM tree.

[0257] Controller 0 can dedupe the fingerprint and logical address in the first partition by merging the LSM tree, and dedupe the corresponding data block from hard disk 134 based on the physical address corresponding to the logical address. For example, controller 0 can delete the fingerprint and logical address of the first data block in the first partition by merging the LSM tree, and delete the first data block from hard disk 134 based on the physical address corresponding to the logical address of the first data block. The specific implementation of deduplication based on the LSM tree can be found in the description of S514 to S516 in the embodiment shown in FIG5 , and will not be repeated here.

[0258] S626: Controller 0 eliminates fingerprints and logical addresses in at least one partition of the inverse mapping table.

[0259] Each partition in the inverse mapping table can be set with corresponding elimination conditions. When a partition in the inverse mapping table meets the corresponding elimination conditions, the metadata such as fingerprints and logical addresses in the partition can be eliminated. Specifically, within each partition, controller 0 can determine the metadata to be eliminated based on the partition capacity and the popularity of the fingerprints corresponding to the data blocks, such as the fingerprints and logical addresses of non-deduplicated metadata and the fingerprints and logical addresses of partially deduplicated metadata, and then eliminate them from the inverse mapping table. This can reduce the metadata size and ensure system performance.

[0260] S628: Controller 0 stores the deduplicated fingerprint and logical address in the fingerprint table.

[0261] S630: Controller 0 obtains feedback information of the current partition adjustment period, so as to be used by controller 0 to determine whether partition adjustment is triggered in the next partition adjustment period.

[0262] In this embodiment, the controller 0 can obtain the above feedback information to assist in adjusting the partitioning decision and improve the partitioning accuracy.

[0263] It should be noted that the above steps S628 to S630 are optional steps of the embodiment of the present application, and the data deduplication method of the embodiment of the present application may not execute the above steps.

[0264] Based on the above description, the data deduplication method of the embodiment of the present application provides an active partitioning mechanism, which converts implicit partitions into active partitions. For example, the inverse mapping table is divided into hot partitions and cold partitions. The hot partitions and cold partitions are allocated corresponding quotas of system resources. Among them, data that is not updated frequently can be allocated sufficient storage space to support deduplication, while data that is updated frequently is frequently invalidated, so the quota resources that can be allocated are relatively small; in this way, the data that can be finally deduplicated can basically be deduplicated instead of being squeezed out and eliminated, thereby improving the deduplication rate; at the same time, data that is difficult to delete can be eliminated, thereby reducing the overall metadata mapping scale and improving system performance.

[0265] Figure 6 illustrates a simple scenario where controller 0 uses a modeling strategy based on dot partitioning. In complex scenarios, controller 0 can also use a modeling strategy based on Gaussian process regression. The modeling process for complex scenarios is described below.

[0266] In machine learning, machine learning algorithms usually predict an optimal output value y based on the input value x, which is used for classification or regression tasks. In this case, y is treated as an ordinary variable. In some cases, the task does not require predicting a function value, but rather giving the posterior probability distribution of this function value, denoted as p. (y|x) In this case, the function value y can be considered a random variable. Gaussian Process Regression (GPR) is a Bayesian modeling of a set of function values ​​for an unknown function (also known as a black-box function) to provide a probability distribution for the function values.

[0267] When using Gaussian process regression to model the partition strategy model, the problem can be described as obtaining the maximum deduplication rate by reasonably allocating the resource proportions of each partition configuration under the premise of limited partition configuration resources. To better describe the problem, this embodiment uses a partition capacity combination S to describe the resource proportions of each partition. S is an n-dimensional array, and the i-th element Si represents the resource proportion of the i-th partition. Under the premise that other factors remain unchanged, the deduplication rate can be regarded as determined by S. At this time, the relationship between the deduplication rate and the partition capacity combination can be defined as f(S). Since f(S) cannot be obtained explicitly, this embodiment uses the modeling strategy of Gaussian process regression to characterize f(S).

[0268] Modeling f(S) based on the Gaussian process regression modeling strategy can include the following stages:

[0269] Initialization phase: Controller 0 randomly generates several partition capacity combinations and adds them to the set Set. Controller 0 then applies these partition capacity combinations to the storage system and, through the storage system operation, determines the deduplication rates for each partition capacity combination. Based on the correspondence between these partition capacity combinations and deduplication rates, a preliminary Gaussian model G is established to characterize f(S).

[0270] Iterative update phase: Gaussian model G recommends a partition capacity combination and adds it to the set Set. Controller 0 applies this partition capacity combination to the storage system and calculates the deduplication rate for this configuration. Controller 0 feeds this partition capacity combination and its corresponding deduplication rate back to the Gaussian model for model update, repeating the iterative update step L times (L is a pre-set number of iterations).

[0271] Output phase: Output a set of partition resource configurations with the highest deduplication rate in the set Set.

[0272] In this way, a Gaussian process regression-based modeling strategy can be implemented in complex scenarios, and a set of partition resource configurations with the highest deduplication rate, namely the target partition combination, can be provided based on the above Gaussian model.

[0273] In order to make the technical solution of the present application clearer and easier to understand, the data deduplication method of an embodiment of the present application is applied to a storage system with a global cache for illustration.

[0274] Referring to the application scenario diagram of the data deduplication method shown in Figure 12, in a distributed storage system with separated storage and computing, a client running on at least one host is connected to multiple storage nodes. Storage nodes such as Node1 and Node2 can start a deduplication service process to execute the data deduplication method. Furthermore, the storage nodes can also launch a LUN service process to assist the deduplication service process in executing the data deduplication method.

[0275] Specifically, a client can send a write request that includes data, which can be divided into at least one data block. For each data block, the storage node can first record the data location information in a forward mapping table before writing the data to disk. The deduplication service process on a storage node, such as Node1, then executes a data deduplication method to identify and remove redundant data from storage media such as disks, significantly increasing available capacity and reducing costs.

[0276] On the basis of data being dropped to disk, the storage node uses asynchronous writing to insert the fingerprint and logical address of the data block into the reverse mapping table in order. The reverse mapping table is the fingerprint metadata management structure introduced in this application. For data with different update frequencies, the storage node uses the data structure of LSM Tree to effectively manage them respectively, and this structure can trigger deduplication by merging tables. After deduplication is completed, the deduplication service process of the storage node sends the deduplicated metadata in the reverse mapping table to the fingerprint table, and updates the physical address information of the deduplicated data in the forward mapping table (that is, the LUN address mapping table) as a fingerprint accordingly.

[0277] Next, referring to Figure 13, a flow chart of a data deduplication method applied to a storage system with a global cache is shown. The global cache has a write cache and a read cache on top. The write cache can be used for LBA heat statistics, eliminating the need for an additional statistics module for heat statistics.

[0278] As shown in Figure 13, a client can send a write request over the network. This write request passes through the global cache's server-side adaptation layer and reaches the write cache. The deduplication service process can then divide the data in the write request into blocks and calculate the fingerprint of each data block. The deduplication service process then queries the fingerprint table. If the data block's fingerprint matches the fingerprint table, it indicates that the storage device stores a data block with the same fingerprint, and the deduplication service process can directly return a write response. If the data block's fingerprint does not match the fingerprint table, it indicates that the storage device does not store a data block with the same fingerprint. The deduplication service process can write the data block to a storage device such as a hard disk and then return a write response.

[0279] The deduplication service process can also sequentially insert the fingerprints and logical addresses of data blocks into the inverse mapping table. Furthermore, the deduplication service process can obtain the LBA heat from the write cache in batches and update the heat of the fingerprint corresponding to the data block based on the LBA heat. The deduplication service process can also obtain workload characteristics such as reuse distance and reuse period, as well as system resource usage information. Based on this system resource usage information, the deduplication service process then performs general feature processing based on these characteristics to obtain structured features.

[0280] The deduplication service process performs partition decision modeling based on the above-mentioned structural features, and then determines the partition capacity combination with the greatest partition benefit based on the partition decision model obtained by the modeling. The deduplication service process partitions the system resources of the inverse mapping table based on the partition capacity combination, specifically partitioning the system resources of the inverse mapping table according to the proportion of resources allocated to hot partitions and the proportion of resources allocated to cold partitions. The deduplication service process constructs the LSM Tree data structure in different partitions to achieve effective management of metadata such as fingerprints and logical addresses. For example, the deduplication service process can trigger deduplication by merging tables based on the LSM tree. The deduplication service process can also decide the metadata that needs to be eliminated based on the partition capacity and the popularity of the fingerprint corresponding to the data block, and eliminate it from the LSM tree of the inverse mapping table, thereby reducing the size of the metadata.

[0281] In this example, after deduplication is complete, the deduplication service process can also send the deduplicated metadata in the inverse mapping table to the fingerprint table, and the LUN service process can correspondingly update the physical address corresponding to the deduplicated data in the address mapping table as a fingerprint. Furthermore, the deduplication service process can also obtain feedback information so that based on this feedback, it can determine whether to trigger partition adjustment in a subsequent stage, such as the next partition adjustment cycle. This can achieve more precise partitioning and thus improve the deduplication rate.

[0282] Based on the data deduplication method provided in the embodiment of the present application, the embodiment of the present application further provides a data deduplication device. Next, the data deduplication device of the embodiment of the present application is introduced from the perspective of functional modularization with reference to the accompanying drawings.

[0283] Referring to the structural diagram of a data deduplication device shown in FIG14 , the device 1400 includes:

[0284] The communication module 1402 is configured to receive a write request, wherein the write request includes a first data block;

[0285] A data writing module 1404 is configured to write the first data block into a storage device;

[0286] The data writing module 1404 is further configured to write metadata of the first data block into a first partition among multiple partitions of a metadata management structure, where the first partition is determined based on characteristics of the first data block, and the metadata of the first data block includes a fingerprint and address information of the first data block;

[0287] The deduplication module 1406 is configured to delete the metadata of the first data block in the first partition when a fingerprint identical to the fingerprint of the first data block exists in the first partition, and delete the first data block from the storage device according to the address information of the first data block.

[0288] The communication module 1402 can be used to implement the content described in S502 in the embodiment shown in Figure 5. The data writing module 1404 is used to implement the content described in S510 in the embodiment shown in Figure 5. The data writing module 1404 is also used to implement the content described in S512 in the embodiment shown in Figure 5. The deduplication module 1406 is used to implement the content described in S514 and S516 in the embodiment shown in Figure 5.

[0289] In some possible implementations, the feature of the first data block is a fingerprint corresponding to the first data block, and the data writing module 1404 is specifically configured to:

[0290] Determining the popularity of the fingerprint corresponding to the first data block;

[0291] determining, according to the popularity of the fingerprint corresponding to the first data block, the first partition among the multiple partitions of the metadata management structure corresponding to the popularity;

[0292] Writing metadata of the first data block into the first partition.

[0293] Among them, the data writing module 1404 determines the popularity of the fingerprint corresponding to the first data block, determines the first partition based on the popularity, and writes the metadata into the first partition. The implementation can be seen in the description of S506 and S512 in the embodiment shown in Figure 5, which will not be repeated here.

[0294] In some possible implementations, the data writing module 1404 is further configured to:

[0295] After writing the metadata of the first data block into the first partition, determining the heat of the logical address of the first data block;

[0296] The heat of the logical address is accumulated to the heat of the fingerprint corresponding to the first data block to update the heat of the fingerprint corresponding to the first data block.

[0297] The implementation of the data writing module 1404 updating the heat can be referred to the relevant description in the embodiment shown in FIG5 , which will not be repeated here.

[0298] In some possible implementations, the write request further includes a second data block, and the popularity of the fingerprint corresponding to the second data block is higher than the popularity of the fingerprint corresponding to the first data block. The write data module 1404 is further configured to:

[0299] The second data block is written into the storage device, and metadata of the second data block is written into a second partition among the multiple partitions of the metadata management structure, where the capacity of the second partition is smaller than the capacity of the first partition.

[0300] The specific implementation of the data writing module 1404 writing the second data block and the metadata of the second data block can refer to the description of writing the first data block and its metadata, which will not be repeated here.

[0301] In some possible implementations, the capacities of the multiple partitions of the metadata management structure are determined based on a partition decision model, which is used to predict the corresponding partition benefits of each partition capacity combination in a preset partition capacity combination after it is applied to the metadata management structure, and to determine the partition capacity combination with the largest partition benefit as the capacity of the multiple partitions of the metadata management structure, wherein the partition benefit is determined based on at least one of a deduplication rate and a partition adjustment cost.

[0302] In some possible implementations, the partition benefit is a deduplication rate, and the preset partition capacity combination includes a first partition capacity combination. The apparatus 1400 further includes a partition module 1408, which is configured to predict a corresponding deduplication rate after the first partition capacity combination is applied to the metadata management structure by:

[0303] Obtaining workload characteristics corresponding to each of the plurality of partitions formed by applying the first partition capacity combination to the metadata management structure;

[0304] Obtaining data distribution corresponding to each partition according to workload characteristics corresponding to each partition;

[0305] The deduplication rate is obtained according to the data distribution corresponding to each partition and the capacity of each partition.

[0306] The specific implementation of the partitioning module 1408 constructing the partitioning decision model can be found in the description of S610 to S616 in the embodiment shown in FIG6 , and will not be repeated here.

[0307] In some possible implementations, the apparatus 1400 further includes a partitioning module 1408, and the partitioning module 1408 is configured to:

[0308] periodically adjusting the capacity of multiple partitions of the metadata management structure;

[0309] When the adjustment time is reached, it is determined whether to adjust the capacities of the plurality of partitions according to the partition revenue corresponding to the period before the adjustment time, the partition capacity combination, or the workload characteristics corresponding to each partition.

[0310] In some possible implementations, the deduplication module 1406 is specifically configured to:

[0311] When a fingerprint identical to the fingerprint of the first data block exists in the first partition and the number of the fingerprints in the first partition reaches a preset threshold, the metadata of the first data block is deleted, and the first data block is deleted from the storage device according to the address information of the first data block.

[0312] The deduplication module 1406 is used to implement the description of S514 and S516 in the embodiment shown in FIG5 , and will not be described in detail here.

[0313] In some possible implementations, the address information of the first data block is a logical address of the first data block, and the data writing module 1404 is further configured to:

[0314] Writing the logical address and the physical address of the first data block into the address mapping table;

[0315] The deduplication module 1406 is specifically configured to:

[0316] Acquire the physical address of the first data block from the address mapping table according to the logical address of the first data block;

[0317] The first data block is found from the storage device according to the physical address, and the first data block is deleted.

[0318] The data writing module 1404 is also used to implement the description of S518 in the embodiment shown in FIG5 , which will not be described in detail here.

[0319] In some possible implementations, the deduplication module 1406 is further configured to:

[0320] After deleting the metadata of the first data block in the first partition, the physical address of the first data block in the address mapping table is modified to the fingerprint of the first data block.

[0321] The deduplication module 1406 is also used to implement the description of S518 in the embodiment shown in Figure 5. No further details will be given here.

[0322] In some possible implementations, the apparatus 1400 further includes:

[0323] The elimination module 1409 is configured to eliminate the metadata in at least one partition in the inverse mapping table when the at least one partition meets an elimination condition.

[0324] The elimination module 1409 is also used to implement the description of the relevant contents of S520 in the embodiment shown in Figure 5. No further details will be given here.

[0325] According to the embodiment of the present application, the data deduplication device 1400 can correspond to executing the method described in the embodiment of the present application, and the above-mentioned and other operations and / or functions of each module / unit of the data deduplication device 1400 are respectively for realizing the corresponding processes of each method in the embodiments shown in Figures 5 and 6. For the sake of brevity, they will not be repeated here.

[0326] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device or a computing device cluster (e.g., a storage system) to execute the above-mentioned data deduplication method.

[0327] The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer or data center to another website, computer or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer program product can be a software installation package. When any of the aforementioned data deduplication methods is required, the computer program product can be downloaded and executed on a computing device or a computing device cluster.

[0328] The descriptions of the processes or structures corresponding to the above figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

Claims

1. A data deduplication method, characterized in that: The method comprises: receiving a write request, wherein the write request includes a first data block; Writing the first data block into a storage device; Writing metadata of the first data block into a first partition of a plurality of partitions of a metadata management structure, wherein the first partition is determined according to a feature of the first data block, and the metadata of the first data block includes a fingerprint and address information of the first data block; When a fingerprint identical to the fingerprint of the first data block exists in the first partition, metadata of the first data block in the first partition is deleted, and the first data block is deleted from the storage device according to address information of the first data block.

2. The method according to claim 1, characterized in that The feature of the first data block is a fingerprint corresponding to the first data block, and writing the metadata of the first data block into a first partition of the multiple partitions of the metadata management structure includes: Determining the heat of the fingerprint corresponding to the first data block; Determine, according to the heat of the fingerprint corresponding to the first data block, the first partition among the multiple partitions of the metadata management structure corresponding to the heat; Writing metadata of the first data block into the first partition.

3. The method according to claim 2, characterized in that After writing the metadata of the first data block into the first partition, the method further includes: Determining the heat of the logical address of the first data block; The temperature of the logical address is accumulated to the temperature of the fingerprint corresponding to the first data block to update the temperature of the fingerprint corresponding to the first data block.

4. The method according to any one of claims 1 to 3, characterized in that: The write request also includes a second data block, the heat of the fingerprint corresponding to the second data block is higher than the heat of the fingerprint corresponding to the first data block, and the method further includes: The second data block is written into the storage device, and metadata of the second data block is written into a second partition among multiple partitions of the metadata management structure, wherein the capacity of the second partition is smaller than the capacity of the first partition.

5. The method according to any one of claims 1 to 4, characterized in that: The capacities of the multiple partitions of the metadata management structure are determined according to a partition decision model, and the partition decision model is used to predict the corresponding partition benefits of each partition capacity combination in the preset partition capacity combination after being applied to the metadata management structure, and determine the partition capacity combination with the largest partition benefit as the capacity of the multiple partitions of the metadata management structure, and the partition benefit is determined according to at least one of the deduplication rate and the partition adjustment cost.

6. The method according to claim 5, characterized in that The partition benefit is a deduplication rate, the preset partition capacity combination includes a first partition capacity combination, and the partition decision model predicts the deduplication rate corresponding to the first partition capacity combination after being applied to the metadata management structure in the following manner: Obtaining workload characteristics corresponding to each of the plurality of partitions formed by applying the first partition capacity combination to the metadata management structure; Obtaining data distribution corresponding to each partition according to workload characteristics corresponding to each partition; The deduplication rate is obtained according to the data distribution corresponding to each partition and the capacity of each partition.

7. The method according to claim 5 or 6, characterized in that: The method further comprises: periodically adjusting the capacity of multiple partitions of the metadata management structure; When the adjustment time is reached, it is determined whether to adjust the capacity of the plurality of partitions according to the partition revenue corresponding to the period before the adjustment time, the partition capacity combination or the workload characteristics corresponding to each partition.

8. The method according to any one of claims 1 to 7, characterized in that: When there is a fingerprint identical to the fingerprint of the first data block in the first partition, deleting metadata of the first data block in the first partition, and deleting the first data block from the storage device according to address information of the first data block, comprises: When there is a fingerprint identical to the fingerprint of the first data block in the first partition and the number of the fingerprints in the first partition reaches a preset threshold, the metadata of the first data block is deleted, and the first data block is deleted from the storage device according to the address information of the first data block.

9. The method according to any one of claims 1 to 8, characterized in that: The address information of the first data block is a logical address of the first data block, and the method further includes: Writing the logical address and the physical address of the first data block into the address mapping table; The deleting the first data block from the storage device according to the address information of the first data block includes: Acquire the physical address of the first data block from the address mapping table according to the logical address of the first data block; The first data block is found from the storage device according to the physical address, and the first data block is deleted.

10. The method according to claim 9, characterized in that After deleting the metadata of the first data block in the first partition, the method further includes: The physical address of the first data block in the address mapping table is modified to the fingerprint of the first data block.

11. The method according to claim 9, characterized in that The method further comprises: When at least one partition in the metadata management structure meets an elimination condition, the metadata in the at least one partition is eliminated.

12. A data deduplication device, characterized in that: The device comprises: A communication module, configured to receive a write request, wherein the write request includes a first data block; A data writing module, used for writing the first data block into a storage device; The data writing module is further used to write the metadata of the first data block into a first partition among the multiple partitions of the metadata management structure, where the first partition is determined according to the characteristics of the first data block, and the metadata of the first data block includes the fingerprint and address information of the first data block; A deduplication module is used to delete the metadata of the first data block in the first partition when a fingerprint identical to the fingerprint of the first data block exists in the first partition, and to delete the first data block from the storage device according to the address information of the first data block.

13. The device according to claim 12, characterized in that The feature of the first data block is a fingerprint corresponding to the first data block, and the data writing module is specifically used for: Determining the heat of the fingerprint corresponding to the first data block; Determine, according to the heat of the fingerprint corresponding to the first data block, the first partition among the multiple partitions of the metadata management structure corresponding to the heat; Writing metadata of the first data block into the first partition.

14. The device according to claim 13, characterized in that The data writing module is also used for: After writing the metadata of the first data block into the first partition, determining the heat of the logical address of the first data block; The temperature of the logical address is accumulated to the temperature of the fingerprint corresponding to the first data block to update the temperature of the fingerprint corresponding to the first data block.

15. The device according to any one of claims 12 to 14, characterized in that The write request also includes a second data block, the heat of the fingerprint corresponding to the second data block is higher than the heat of the fingerprint corresponding to the first data block, and the write data module is further used for: The second data block is written into the storage device, and metadata of the second data block is written into a second partition among multiple partitions of the metadata management structure, wherein the capacity of the second partition is smaller than the capacity of the first partition.

16. The device according to any one of claims 12 to 15, characterized in that The capacities of the multiple partitions of the metadata management structure are determined according to a partition decision model, and the partition decision model is used to predict the corresponding partition benefits of each partition capacity combination in the preset partition capacity combination after being applied to the metadata management structure, and determine the partition capacity combination with the largest partition benefit as the capacity of the multiple partitions of the metadata management structure, and the partition benefit is determined according to at least one of the deduplication rate and the partition adjustment cost.

17. The device according to claim 16, characterized in that The partition benefit is a deduplication rate, the preset partition capacity combination includes a first partition capacity combination, and the partition decision model predicts the deduplication rate corresponding to the first partition capacity combination after being applied to the metadata management structure in the following manner: Obtaining workload characteristics corresponding to each of the plurality of partitions formed by applying the first partition capacity combination to the metadata management structure; Obtaining data distribution corresponding to each partition according to workload characteristics corresponding to each partition; The deduplication rate is obtained according to the data distribution corresponding to each partition and the capacity of each partition.

18. The device according to claim 16 or 17, characterized in that The device further comprises a partition module, wherein the partition module is configured to: periodically adjusting the capacity of multiple partitions of the metadata management structure; When the adjustment time is reached, it is determined whether to adjust the capacity of the plurality of partitions according to the partition income corresponding to the period before the adjustment time, the partition capacity combination or the workload characteristics corresponding to each partition.

19. The device according to any one of claims 12 to 18, characterized in that The deduplication module is specifically used for: When a fingerprint identical to the fingerprint of the first data block exists in the first partition and the number of the fingerprints in the first partition reaches a preset threshold, the metadata of the first data block is deleted, and the first data block is deleted from the storage device according to the address information of the first data block.

20. The device according to any one of claims 12 to 19, characterized in that The address information of the first data block is a logical address of the first data block, and the data writing module is further used for: Writing the logical address and the physical address of the first data block into the address mapping table; The deduplication module is specifically used for: Acquire the physical address of the first data block from the address mapping table according to the logical address of the first data block; The first data block is found from the storage device according to the physical address, and the first data block is deleted.

21. The device according to claim 20, characterized in that The deduplication module is also used for: After deleting the metadata of the first data block in the first partition, the physical address of the first data block in the address mapping table is modified to the fingerprint of the first data block.

22. The device according to claim 20, characterized in that The device also includes: The elimination module is used to eliminate the metadata in at least one partition when at least one partition in the inverse mapping table meets the elimination condition.

23. A storage system, characterized in that: The storage system includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the storage system executes the method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 11.

25. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 11.

Images