A data deduplication method and related system

Abstract

This application provides a data deduplication method, comprising: in response to a data deduplication request, determining multiple duplicate blocks from data blocks stored in persistent media; determining a block to be deduplicated from the multiple duplicate blocks according to the deduplication priority of the multiple duplicate blocks, wherein the deduplication priority of the duplicate blocks is determined based on the attributes of the duplicate blocks; and then performing deduplication on the block to be deduplicated. After the block to be deduplicated is deduplicated, the fragmentation degree of the data in the persistent media is less than the fragmentation degree of the data in the persistent media after the data blocks other than the block to be deduplicated are deduplicated. This method sets a deduplication priority for data blocks by analyzing the different degrees of fragmentation caused by the deduplication of data blocks, and selects a block to be deduplicated from multiple duplicate blocks according to the deduplication priority, thereby reducing the degree of read fragmentation and improving read efficiency.

Description

[0001] This application is a divisional application. The original application has the application number 202311077281.6 and the original application date is August 24, 2023. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] This application relates to the field of storage technology, and in particular to a data deduplication method, apparatus, storage system, computer-readable storage medium, and computer program product. Background Technology

[0003] With the advent of the information age, data is experiencing explosive growth. To fully utilize data and unlock its value, storage systems are typically needed to store it. However, to meet the ever-increasing demand for data storage, large-capacity storage systems are required, resulting in persistently high storage costs.

[0004] Considering that storage systems often contain a large amount of duplicate data—for example, 50% of the data in main memory is duplicated—data deduplication technology can be used to reduce the amount of data stored, thereby reducing data storage costs. The data deduplication process can include dividing the data into blocks, calculating the fingerprint (FP) of each block based on its content, identifying duplicate blocks based on their fingerprints, and deleting those duplicates. This achieves the goal of reducing storage costs.

[0005] However, data deduplication can cause data that was originally stored contiguously to be distributed across different areas of the storage system, such as different nodes in a distributed storage system. When reading data, a single continuous input / output (IO) operation can degenerate into multiple random read IO operations. In other words, data deduplication can lead to a significant "read fragmentation" problem, greatly impacting read performance. Summary of the Invention

[0006] This application provides a data deduplication method that introduces deduplication priority and selects blocks to be deduplicated from multiple duplicate blocks based on the priority, thereby reducing read fragmentation and improving read efficiency. This application also provides an apparatus, storage system, computer-readable storage medium, and computer program product corresponding to the above method.

[0007] Firstly, this application provides a data deduplication method. This method can be executed by a storage system. The storage system can be a centralized storage system, including a centralized storage system with integrated disk and controller, a centralized storage system with separate disk and controller, or the storage system can also be a distributed storage system, including but not limited to a distributed storage system with integrated storage and computing, or a distributed storage system with separate storage and computing.

[0008] Specifically, in response to a data deduplication request, the storage system identifies multiple duplicate blocks from the data blocks stored in the persistent media. Based on the deduplication priority of these duplicate blocks, it determines the block to be deduplicated from among them. The deduplication priority of the duplicate blocks is determined based on their attributes. The storage system then performs deduplication on the block to be deduplicated. After the block to be deduplicated is deleted, the degree of data fragmentation in the persistent media is less than the degree of data fragmentation in the persistent media after deduplicating the data blocks other than the block to be deduplicated from the multiple duplicate blocks.

[0009] In this method, the storage can identify the differences in deduplication opportunities among different data blocks, perceive the deduplication priority of data blocks, and select blocks to be deduplicated from multiple duplicate blocks based on the deduplication priority. This enables targeted deduplication, which can reduce the degree of read fragmentation when reading data later and improve read performance.

[0010] In some possible implementations, the deduplication block includes multiple blocks that satisfy aggregation conditions. These aggregation conditions can be logical address contiguousness, logical address intervals less than a set length, or logical address intervals equal to a step size. A deduplication block satisfying these aggregation conditions can be considered to have locality (in other words, this data block has a high probability of being read along with other data blocks in subsequent reads). Based on this, when performing deduplication on the aforementioned blocks, the storage system can aggregate multiple blocks satisfying the aggregation conditions and write them to the persistent medium, and then perform deduplication on these multiple blocks satisfying the aggregation conditions.

[0011] This method also supports the aggregation of localized deduplication blocks (multiple deduplication blocks that meet the aggregation conditions) and batch writing them to the persistent medium (this process is also called local aggregation reorganization), thereby further reducing the fragmentation of data in the persistent medium and improving read performance.

[0012] In some possible implementations, multiple deduplication blocks that meet the aggregation conditions are determined as follows: multiple deduplication blocks determined from the multiple duplicate blocks are sorted according to their logical addresses, and from the sorted multiple deduplication blocks, the multiple deduplication blocks that meet the aggregation conditions are determined.

[0013] This method sorts the blocks to be deduplicated according to their logical addresses, identifies the blocks with locality, aggregates them, and writes them into the persistent medium. This enables the blocks with locality to be deduplicated to be stored contiguously on the persistent medium (physical space), thereby reducing the number of random I / O operations and the degree of fragmentation during reading.

[0014] In some possible implementations, the attributes of the duplicate block include at least one of the following: the number of blocks routed to the same node as the duplicate block, and the expected read granularity of the duplicate block. This method, by analyzing factors such as the number of blocks routed to the same node as the duplicate block and the expected read granularity of the duplicate block, which affect the degree of fragmentation during subsequent data reads, can accurately determine the deduplication priority of duplicate blocks, providing a reference for data deduplication.

[0015] In some possible implementations, the deduplication priority of the duplicate block is determined as follows: when the expected read granularity of the duplicate block is less than a preset value, the priority of the duplicate block is determined to be the first priority; when the expected read granularity of the duplicate block is greater than or equal to the preset value, the priority of the duplicate block is determined to be the second priority based on the number of blocks routed to the same node as the duplicate block. The first priority is higher than the second priority.

[0016] Among them, duplicate blocks with smaller expected read granularity are usually not fragmented or have a smaller degree of fragmentation when they are deleted. Based on this, this type of duplicate block can be deleted first. Duplicate blocks with larger expected read granularity can be fragmented when they are deleted. Based on this, the deletion priority can be determined by combining the number of blocks that are routed to the same node as the duplicate block.

[0017] This method analyzes the impact of deleting duplicate blocks on fragmentation from different dimensions, prioritizing the deletion of smaller duplicate blocks, thereby reducing fragmentation and improving read performance.

[0018] In some possible implementations, the repeating block includes a first repeating block and a second repeating block. When the number of blocks routed to the same node as the first repeating block is greater than the number of blocks routed to the same node as the second repeating block, the priority of the first repeating block is higher than the priority of the second repeating block.

[0019] This method configures the deduplication priority to be positively correlated with the number of blocks routed to the same node. This allows data blocks that are prone to read amplification to be deduplicated first, thus ensuring subsequent read performance.

[0020] In some possible implementations, the storage system can also obtain a target deduplication rate, perform pre-deduplication on the duplicate blocks, determine the deduplication rate when the duplicate blocks are deduplicated according to a candidate deduplication ratio, and adjust the candidate deduplication ratio when the deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio is less than the target deduplication rate, until a target deduplication ratio is determined that makes the deduplication rate reach the target deduplication rate. Accordingly, the storage system can determine the blocks to be deduplicated from the multiple duplicate blocks based on the deduplication priority of the multiple duplicate blocks and the target deduplication ratio.

[0021] In this way, the deduplication ratio can be adaptively adjusted to determine the target deduplication ratio to achieve the desired deduplication rate. The blocks to be deduplicated are selected and deduplicated according to the target deduplication ratio, and the deduplication rate can be dynamically returned to the achievable deduplication rate set by the system.

[0022] In some possible implementations, before identifying multiple duplicate blocks from the data blocks stored in persistent media, the storage system may also receive write requests and query a cache based on the fingerprint of the data block in the write request. The cache stores the fingerprints of deduplicated blocks. If the fingerprint of a data block is not found in the cache, the storage system may write the data block to persistent media.

[0023] This method introduces a caching mechanism. When writing data, it first checks whether the fingerprint of the data block is found in the cache. If it is not found, the data block is then written to the persistent medium. This can filter duplicate data, reduce the pressure of background deduplication, and thus avoid the back pressure on the front-end read and write performance caused by background deduplication read and write amplification, thereby ensuring the performance of the front-end read and write.

[0024] In some possible implementations, when the write request includes a target data block whose fingerprint matches in the cache, the storage system can further determine the frequency of the target data block based on the reuse frequency of the fingerprint or the reuse distance. Frequency includes local frequency or global frequency. The storage system can determine the target caching algorithm to be used for the target data block based on its frequency, and cache the target data block according to the target caching algorithm.

[0025] This method analyzes the frequency of data blocks based on their reuse frequency and reuse distance, determines appropriate caching strategies based on the frequency, improves cache hit rate, and further enhances caching efficiency.

[0026] Secondly, this application provides a data deduplication apparatus. The apparatus includes:

[0027] The deduplication block selection module is used to respond to a data deduplication request, determine multiple duplicate blocks from the data blocks stored in the persistent medium, and determine the block to be deduplicated from the multiple duplicate blocks according to the deduplication priority of the multiple duplicate blocks, wherein the deduplication priority of the duplicate block is determined based on the attributes of the duplicate block.

[0028] The deduplication execution module is used to perform deduplication on the block to be deduplicated. After the block to be deduplicated is deduplicated, the degree of data fragmentation in the persistent medium is less than the degree of data fragmentation in the persistent medium after the data blocks other than the block to be deduplicated among the multiple duplicate blocks are deduplicated.

[0029] In some possible implementations, the deduplication block includes multiple deduplication blocks that satisfy the aggregation condition, and the deduplication execution module is specifically used for:

[0030] The multiple deduplication blocks that meet the aggregation conditions are aggregated and then written to the persistent medium;

[0031] The multiple blocks that meet the aggregation conditions are then deduplicated.

[0032] In some possible implementations, the deduplication execution module is specifically used to determine multiple blocks to be deduplicated that satisfy the aggregation conditions in the following manner:

[0033] The multiple blocks to be deleted, determined from the multiple duplicate blocks, are sorted according to their logical addresses;

[0034] From the sorted multiple deduplication blocks, determine the multiple deduplication blocks that meet the aggregation conditions.

[0035] In some possible implementations, the attributes of the repeating block include at least one of the following: the number of blocks routed to the same node as the repeating block, and the expected read granularity of the repeating block.

[0036] In some possible implementations, the deduplication priority of the duplicate block is determined in the following way:

[0037] When the expected read granularity of the repeated block is less than a preset value, the priority of the repeated block is determined to be the first priority.

[0038] When the expected read granularity of the duplicate block is greater than or equal to the preset value, the priority of the duplicate block is determined as the second priority based on the number of blocks routed to the same node as the duplicate block;

[0039] The first priority is higher than the second priority.

[0040] In some possible implementations, the repeating block includes a first repeating block and a second repeating block. When the number of blocks routed to the same node as the first repeating block is greater than the number of blocks routed to the same node as the second repeating block, the priority of the first repeating block is higher than the priority of the second repeating block.

[0041] In some possible implementations, the deduplication block selection module is further used for:

[0042] Obtain the target deduplication rate;

[0043] Pre-deletion is performed on the duplicate blocks to determine the deletion rate when the duplicate blocks are deleted according to the candidate deletion ratio;

[0044] When the deduplication rate of the duplicate block is less than the target deduplication rate when it is deduplicated according to the candidate deduplication ratio, the candidate deduplication ratio is adjusted until a target deduplication ratio is determined that makes the deduplication rate reach the target deduplication rate.

[0045] The deduplication block selection module is specifically used for:

[0046] Based on the deduplication priority of the multiple duplicate blocks and the target deduplication ratio, the blocks to be deduplicated are determined from the multiple duplicate blocks.

[0047] In some possible implementations, the device further includes:

[0048] The write module is used to receive write requests, query the cache based on the fingerprint of the data block in the write request, the cache stores the fingerprints of deletable blocks, and when the fingerprint of the data block is not found in the cache, the data block is written to the persistent medium.

[0049] In some possible implementations, the writing module is also used for:

[0050] When the write request includes a target data block whose fingerprint is hit in the cache, the frequency of the target data block is determined based on the reuse frequency of the fingerprint or the reuse distance. The frequency includes local frequency or global frequency. Based on the frequency of the target data block, the target caching algorithm to be used for the target data block is determined, and the target data block is cached according to the target caching algorithm.

[0051] Thirdly, this application provides a storage system. The storage system includes a controller and a memory. The controller and the memory communicate with each other. The controller is used to execute instructions stored in the memory to cause the storage system to perform the data deduplication method as described in the first aspect or any implementation thereof.

[0052] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct a storage system to perform the data deduplication method described in the first aspect or any implementation thereof.

[0053] Fifthly, this application provides a computer program product containing instructions that, when run on a storage system, causes the storage system to execute the data deduplication method described in the first aspect or any implementation thereof.

[0054] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0055] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0056] Figure 1AA schematic diagram of the architecture of a centralized storage system with disk-controller separation provided for an embodiment of this application;

[0057] Figure 1B A schematic diagram of the architecture of a centralized storage system integrating disk control, provided for an embodiment of this application;

[0058] Figure 2A A schematic diagram of the architecture of a distributed storage system integrating in-memory computing provided for an embodiment of this application;

[0059] Figure 2B A schematic diagram of the architecture of a distributed storage system with storage-compute separation provided in an embodiment of this application;

[0060] Figure 3 A flowchart of a data deduplication method provided in an embodiment of this application;

[0061] Figure 4 A schematic diagram of a data block routing provided in an embodiment of this application;

[0062] Figure 5 This is a schematic diagram illustrating how to determine the deduplication priority and adjust the deduplication ratio to achieve an achievable deduplication rate, as provided in an embodiment of this application.

[0063] Figure 6 A schematic diagram illustrating the process of performing deduplication on a block to be deduplicated, provided as an embodiment of this application;

[0064] Figure 7 This application provides a schematic diagram illustrating the distribution of blocks to be deleted before and after deduplication.

[0065] Figure 8 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0066] Figure 9 A schematic diagram illustrating globally frequent and locally frequent data in an embodiment of this application;

[0067] Figure 10 This is a schematic diagram of a cache algorithm selection process provided in an embodiment of this application;

[0068] Figure 11 This is a schematic diagram of a data deduplication device provided in an embodiment of this application. Detailed Implementation

[0069] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0070] First, some technical terms involved in the embodiments of this application will be introduced.

[0071] In the field of data deduplication technology, a fingerprint (FP) is a value obtained by hashing (encrypted or unencrypted) data content. Fingerprints can be used to represent data content.

[0072] Data deduplication typically involves determining the fingerprints of data blocks after data is segmented. Duplicate blocks are identified and deleted by comparing these fingerprints, thus reducing data size and storage costs. Data deduplication can be categorized into pre-deduplication and post-deduplication based on the timing. Post-deduplication, also known as background deduplication, refers to deduplication performed after the data has been persistently stored. Background deduplication has lower write latency because it does not require fingerprint comparison during write operations.

[0073] The data deduplication method provided in this application can be applied to storage systems. These storage systems can be logically categorized as follows: primary storage and backup storage. Primary storage serves as a storage pool or hard drive for upper-layer applications and can be accessed by applications deployed via virtual machines (VMs). Backup storage acts as a repository, storing images, backup disks, or snapshots, and is typically not directly accessible to VMs. Primary and backup storage can be physically separate storage systems, or the same storage system can serve as both primary and backup storage. Storage systems can also be categorized based on application scenarios: file storage, database storage (including row-oriented or column-oriented databases), and big data storage.

[0074] The aforementioned storage system (including primary storage and backup storage) can adopt either local storage or cloud storage models. Local storage refers to storing data on one's own physical storage devices, while cloud storage refers to storing data in a cloud environment to support network access. A cloud environment refers to a cluster built by a cloud service provider (also known as a cloud vendor) to provide computing, storage, and network resources. This storage system supports block storage, object storage, or file storage.

[0075] Storage systems can employ either a centralized or a distributed storage architecture; in other words, a storage system can be either a centralized or a distributed storage system. Examples are provided below to illustrate each approach.

[0076] A centralized storage system refers to a system consisting of one or more master devices forming a central node. Data is centrally stored in this central node, and all data processing operations of the entire system are centrally deployed on this central node. In other words, in a centralized storage system, terminals or clients are only responsible for data input and output, while data storage and control processing are entirely handled by the central node. The biggest advantage of a centralized system is its simple deployment structure; there's no need to consider how to deploy services across multiple nodes, and therefore no need to worry about distributed collaboration between multiple nodes.

[0077] For ease of understanding, Figure 1A An architecture diagram of a centralized storage system is shown. Figure 1A In the illustrated application scenario, users access data through applications. The computer running these applications is called an "application server." Application server 100 can be a physical machine or a virtual machine (VM). Physical machines (physical application servers) include, but are not limited to, desktop computers, servers, laptops, and mobile devices. The application server accesses the storage system to access data via a fiber optic switch 110. However, switch 110 is only an optional device; application server 100 can also communicate directly with storage system 120 via the network. Alternatively, fiber optic switch 110 can be replaced with an Ethernet switch, an InfiniBand (IB) switch, or an RDMA (RDMA over Converged Ethernet, RoCE) switch, etc.

[0078] Figure 1A The storage system 120 shown is a centralized storage system. A key feature of a centralized storage system is a unified entry point through which all data from external devices passes. This entry point is the engine 121 of the centralized storage system. The engine 121 is the most crucial component of the centralized storage system, and many advanced functions of the storage system are implemented within it.

[0079] like Figure 1A As shown, engine 121 contains one or more controllers. Figure 1ALet's take an engine with two controllers as an example. Controller 0 and Controller 1 have a mirror channel. When Controller 0 writes data to its memory 124, it can send a copy of that data to Controller 1 through the mirror channel. Controller 1 then stores the copy in its local memory 124. Thus, Controller 0 and Controller 1 act as backups for each other. When Controller 0 fails, Controller 1 can take over its operations, and vice versa, preventing hardware failures from rendering the entire storage system 120 unavailable. When four controllers are deployed in engine 121, any two controllers have a mirror channel, thus any two controllers act as backups for each other.

[0080] Engine 121 also includes a front-end interface 125 and a back-end interface 126. The front-end interface 125 is used to communicate with the application server 100 to provide storage services to the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, engine 121 can connect to more hard disks 134, thus forming a very large storage resource pool.

[0081] In terms of hardware, such as Figure 1A As shown, controller 0 includes at least processor 123 and memory 124. Processor 123 is a central processing unit (CPU) used to process data access requests from outside the storage system (server or other storage systems), and also to process requests generated within the storage system. For example, when processor 123 receives write data requests from application server 100 through front-end port 125, it temporarily stores the data in these write data requests in memory 124. When the total amount of data in memory 124 reaches a certain threshold, processor 123 sends the data stored in memory 124 to hard disk 134 for persistent storage through back-end port.

[0082] Memory 124 refers to internal memory that directly exchanges data with the processor. It can read and write data at any time and at high speed, serving as temporary data storage for the operating system or other running programs. Memory includes at least two types of memory, such as random access memory (RAM) and read-only memory (ROM). For example, RAM can be Dynamic Random Access Memory (DRAM) or Storage Class Memory (SCM), while ROM can be Programmable Read Only Memory (PROM) or Erasable Programmable Read Only Memory (EPROM), etc.

[0083] The memory 124 stores software programs, and the processor 123 runs these programs to manage the hard drive. For example, the hard drive can be abstracted into a storage resource pool and then divided into LUNs for server use. Here, the LUN is essentially the hard drive seen on the server. Of course, some centralized storage systems are also file servers themselves, providing shared file services to the server.

[0084] Controller 1 (and others) Figure 1A The hardware components and software structure of the controller (not shown) are similar to those of controller 0, and will not be described in detail here.

[0085] Figure 1A This illustrates a centralized storage system with disk controller separation. In this system, engine 121 may not have hard drive bays; hard drive 134 needs to be placed in hard drive enclosure 130, and back-end interface 126 communicates with hard drive enclosure 130. Back-end interface 126 exists in the form of an adapter card within engine 121, and two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect multiple hard drive enclosures. Alternatively, the adapter card can be integrated onto the motherboard, in which case it can communicate with processor 123 via the PCIe bus.

[0086] It should be noted that, Figure 1A Only one engine 121 is shown in the figure. However, in actual applications, the storage system may contain two or more engines 121, and redundancy or load balancing may be performed between multiple engines 121.

[0087] The hard disk enclosure 130 includes a control unit 131 and several hard disks 134. The control unit 131 can have various forms. Typically, there can be one, two, or more control units 131. When the hard disk enclosure 130 contains at least two control units 131, there can be a hierarchical relationship between the hard disks 134 and the control units 131. The functions of the control unit 131 can be offloaded to the network interface card 104. In this embodiment, the hard disk enclosure 130 does not contain a control unit 131; instead, the network interface card 104 performs data read / write, address translation, and other computational functions. In this case, the network interface card 104 is a smart network interface card. It can contain a CPU and memory. In some application scenarios, the network interface card 104 may also have persistent memory media, such as persistent memory (PM), non-volatile random access memory (NVRAM), or phase change memory (PCM). The CPU is used to perform address translation and data read / write operations. The memory is used to temporarily store data that will be written to the hard disk 134, or to read data from the hard disk 134 that will be sent to the controller.

[0088] Centralized storage systems can also be integrated disk-controller architectures. In an integrated disk-controller architecture, engine 121 has hard drive bays, and hard drives 134 can be directly deployed within engine 121. The back-end interface 126 is an optional configuration; when the system's storage space is insufficient, more hard drives or hard drive enclosures can be connected via back-end interface 126. For example... Figure 1B As shown, engine 121 can connect to more hard drives 134 through back-end interface 126.

[0089] A distributed storage system is a system that distributes data across multiple independent storage nodes. Distributed storage systems employ a scalable system architecture, utilizing multiple storage nodes to share the storage load. This not only improves system reliability, availability, and access efficiency but also facilitates expansion.

[0090] Distributed storage systems can adopt either an integrated storage-compute architecture or a separate storage-compute architecture. These will be explained in detail below.

[0091] like Figure 2A As shown, the in-memory computing distributed storage system includes a storage cluster. The storage cluster includes one or more servers 110 ( Figure 2AThe diagram shows three servers 110 (but is not limited to three servers 110), which can communicate with each other. A server 110 is a device with both computing and storage capabilities, such as a server or desktop computer. For example, an ARM server or an x86 server can be used as a server 110. In terms of hardware, a server 110 includes at least a processor 112, memory 113, a network interface card (NIC) 114, and a hard disk 105. The processor 112, memory 113, NIC 114, and hard disk 105 are connected via a bus. The processor 112 is a central processing unit (CPU) used to process data access requests from outside the server 110 (application servers or other servers 110), and also to process requests generated internally within the server 110. In addition, the processor 112 is used for data computation or processing, such as metadata management, deduplication, data compression, data verification, virtualization of storage space, and address translation.

[0092] like Figure 2B As shown, the distributed storage cluster with separate storage and compute capabilities includes a compute cluster (also called a compute node cluster) and a storage cluster (also called a storage node cluster). The compute nodes 110 can communicate with each other. A compute node 110 is a computing device, such as a server, desktop computer, or the controller of a storage array. In terms of hardware, such as... Figure 2B As shown, compute node 110 includes at least a processor 112, memory 113, and a network interface card (NIC) 114. The processor 112 is a central processing unit (CPU) used to handle data access requests from outside the compute node 110 or requests generated internally within the compute node 110. When the total amount of data in memory 113 reaches a certain threshold, the processor 112 sends the data stored in memory 113 to storage node 100 for persistent storage. In addition, the processor 112 is also used for data computation or processing, such as metadata management, deduplication, data compression, virtualization of storage space, and address translation.

[0093] Any compute node 110 can access any storage node 100 in the storage node cluster via the network. The storage node cluster includes multiple storage nodes 100 ( Figure 2BThe diagram shows three storage nodes 100, but is not limited to three storage nodes 100. Each storage node 100 includes one or more controllers 101, a network interface card (NIC) 104, and multiple hard disks 105. The NIC 104 is used to communicate with the compute node 110. The hard disks 105 are used to store data and can be disks or other types of storage media, such as solid-state drives (SSDs) or shingled magnetic recording (SMR) hard disks. The controllers 101 are used to write data to or read data from the hard disks 105 according to read / write data requests sent by the compute node 110. During the read / write process, the controllers 101 need to convert the addresses carried in the read / write data requests into addresses that the hard disks can recognize. Therefore, the controllers 101 also have some basic computing functions.

[0094] It should be noted that the above is merely an example of a storage system. In other possible implementations of the embodiments of this application, the storage system may also be a distributed storage system or a centralized storage system with other architectures. For example, the storage system may also be a fully converged distributed storage system.

[0095] When persisting data, the storage system can divide the data into multiple data blocks. These multiple data blocks can be routed to the persistence media of the corresponding nodes for persistent storage based on fingerprints. Then, the storage system can identify duplicate blocks in the background and perform deduplication on the duplicate blocks after identification to reduce storage costs.

[0096] However, deduplication can cause data that could have been stored contiguously to be scattered across different nodes or partitions. Subsequent data retrieval, where a single sequential read I / O might have been possible, often requires multiple random read I / Os (across nodes or partitions) after deduplication. This results in a significant "read fragmentation" problem.

[0097] In view of this, this application provides a data deduplication method. Addressing the read fragmentation problem, this method sets deduplication priorities for data blocks by analyzing the different degrees of fragmentation resulting from deduplication. Based on these priorities, it selects blocks to be deduplicated from multiple duplicate blocks, thereby reducing read fragmentation and improving read efficiency. Furthermore, this method can also identify the locality of duplicate blocks (if a data block has a high probability of being read along with other data blocks in subsequent reads, it can be considered to have locality), and aggregate these locally local duplicate blocks through reordering, then persist them in batches, further reducing read fragmentation and improving read efficiency.

[0098] To make the technical solution of this application clearer and easier to understand, the data deduplication method of this application will be described below with reference to embodiments.

[0099] See Figure 3 The flowchart shown illustrates a data deduplication method, which includes the following steps:

[0100] S302. In response to a data deduplication request, the storage system identifies multiple duplicate blocks from the data blocks stored on the persistent media.

[0101] Data deduplication requests are used to trigger data deduplication. In some possible implementations, a data deduplication request can be generated by an upper-layer application of the storage system in response to a user-triggered deduplication operation. For example, a user can enable data deduplication in the application's settings interface, and the application generates a data deduplication request in response to this operation and sends the request to the storage system. In other possible implementations, a data deduplication request can be automatically generated by the storage system based on pre-set deduplication triggering conditions. These conditions may include reaching a set deduplication time, the time interval since the last deduplication reaching a set duration, the remaining capacity of the storage system being less than a set capacity, or the amount of duplicate data in the storage system reaching a set proportion.

[0102] Specifically, when a storage system receives a data deduplication request, it can first identify duplicate blocks with identical content from the data blocks stored on the persistent media, and then delete these duplicate blocks. Since the fingerprint of a data block is calculated from its content, if the data block content is the same, the data block fingerprint will also be the same. Therefore, the storage system can identify duplicate blocks by comparing the fingerprints of the data blocks.

[0103] In some possible implementations, the storage system can identify multiple duplicate blocks online. Specifically, when writing data blocks to persistent media, the storage system can identify duplicate blocks in the persistent media based on the fingerprints of the data blocks being written to the persistent media this time and the fingerprints of data blocks written to the persistent media previously. Specifically, the storage system can maintain fingerprint metadata, which includes the fingerprints of data blocks already written to the persistent media. The storage system can identify duplicate blocks in the persistent media by comparing the fingerprint of the data block being written to the persistent media with the fingerprints in the fingerprint metadata. For example, if the fingerprint of the data block being written to the persistent media this time includes FP2, and the fingerprint metadata includes the fingerprints FP1, FP2, and FP3, then the data block being written to the persistent media this time and a previously written data block with fingerprint FP2 can be identified as duplicate blocks.

[0104] S304. The storage system determines the block to be deleted from multiple duplicate blocks based on the deduplication priority of the multiple duplicate blocks.

[0105] The storage system can perform online theoretical analysis of the fragmentation caused by deduplication of different data blocks (such as duplicate blocks) and set deduplication priorities for different data blocks (such as duplicate blocks) accordingly. Data blocks with higher deduplication priorities are deleted before those with lower priorities. It should be noted that the storage system can determine the deduplication priority when a data block identified as a duplicate is written to persistent media. The storage system can store the deduplication priority of this data block and select blocks to be deduplicated based on this priority during subsequent deduplication processes. Alternatively, the storage system can determine the deduplication priority of duplicate blocks in real time during the deduplication process.

[0106] The deduplication priority of a data block (e.g., a data block identified as a duplicate) can be determined based on the data block's attributes. In some possible implementations, the data block's attributes may include at least one of the expected read granularity and the number of blocks routed to the same node as the data block. Based on this, for duplicate blocks, the storage system can determine the deduplication priority of the duplicate block based on the expected read granularity of the duplicate block and / or the number of blocks routed to the same node as the duplicate block. The number of blocks routed to the same node as the duplicate block refers to the number of blocks in the same write request (write I / O) that are routed to the same node as the duplicate block. For example, if data in a write request is divided into data blocks 1 to 5, where data block 1 is a duplicate block, and data blocks 1, 3, and 5 are routed to node 1, then the number of blocks routed to the same node as the duplicate block 1 is 2.

[0107] In practical implementation, the storage system (e.g., storage nodes within the storage system) can obtain the expected read granularity of duplicate blocks. When the expected read granularity of a duplicate block is less than a preset value, it indicates that the duplicate block is a data block for which deduplication will not cause read amplification, or that the duplicate block is a data block with a low degree of fragmentation after deduplication. The storage system can determine the priority of the duplicate block as the first priority. In some examples, the first priority can be the highest priority. When the expected read granularity of a duplicate block is greater than or equal to the preset value, it indicates that the duplicate block is a data block for which deduplication will cause read amplification, or that the duplicate block is a data block with a high degree of fragmentation after deduplication. The storage system can determine the priority of the duplicate block as the second priority based on the number of blocks routed to the same node as the duplicate block. The first priority is higher than the second priority. It should be noted that the above preset value can be set based on experience, for example, it can be set to be equal to the deduplication granularity. In some examples, the preset value can be set to 4KB.

[0108] Expected read granularity refers to the anticipated granularity of subsequent reads. In some possible implementations, the expected read granularity can be equal to the granularity of the write request (or write I / O) to which the data block belongs. For example, if the data size in a write request is 1 megabyte (MB), then after the data blocks are written to persistent media, if any data block is identified as a duplicate, the expected read granularity of that duplicate block can be 1 MB. In other possible implementations, the expected read granularity can also be determined through statistical analysis of historical data. For example, the storage system can determine the expected read granularity based on the number of times previously written data blocks with the same fingerprint have been accessed and the read I / O size during each access. Specifically, the storage system can determine the average read I / O size based on the number of times data blocks with the same fingerprint have been accessed and the read I / O size during each access, and the storage system can use this average value as the expected read granularity. Alternatively, the storage system can also determine the median value of the read I / O size during each access, and the storage system can use this median value as the expected read granularity.

[0109] Furthermore, the second priority can be positively correlated with the number of blocks that the duplicate block is routed to the same node. Specifically, the duplicate block can include a first duplicate block and a second duplicate block. When the number of blocks that the first duplicate block is routed to the same node is greater than the number of blocks that the second duplicate block is routed to the same node, the priority of the first duplicate block is higher than the priority of the second duplicate block.

[0110] To facilitate understanding, examples are provided below. Figure 4 As shown, the data flow includes two data blocks with fingerprint FP1 (denoted as data blocks 1 and 2), two data blocks with fingerprint FP2 (denoted as data blocks 3 and 5), and two data blocks with fingerprint FP3 (denoted as data blocks 4 and 6). Data blocks 1 and 2, and data blocks 3 and 5 are routed to node 1, while data blocks 4 and 6 are routed to node 2. Data blocks 1, 2, 3, 5, 4, and 6 are identified as duplicate blocks. This example assumes that the expected read granularity of the above data blocks is greater than 4KB. The storage system can determine the priority of duplicate blocks based on the number of blocks routed to the same node as the duplicate blocks. For data blocks 1, 2, 3, and 5, the number of blocks routed to the same node is 3; for data blocks 4 and 6, the number of blocks routed to the same node is 1. Therefore, it can be determined that the deduplication priority of data blocks 1, 2, 3, and 5 is higher than that of data blocks 4 and 6.

[0111] When selecting blocks to be deduplicated, the storage system can prioritize duplicate blocks with higher deduplication priority. In some possible implementations, the storage system can sort duplicate blocks in descending order of their deduplication priority, and then select the top n or top m% of duplicate blocks as the blocks to be deduplicated. In other possible implementations, the storage system can also select duplicate blocks with a deduplication priority higher than a preset priority as the blocks to be deduplicated.

[0112] Furthermore, the storage system can also obtain the target deduplication rate, which refers to the achievable deduplication rate of the storage system. The storage system can perform pre-deduplication on duplicate blocks to determine the deduplication rate when duplicate blocks are deduplicated according to the candidate deduplication ratio. If the deduplication rate when deduplicated according to the candidate deduplication ratio is less than the target deduplication rate, the storage system can adjust the candidate deduplication ratio until a target deduplication ratio is determined that makes the deduplication rate reach the target deduplication rate. Figure 5 As shown, the storage system can input the fingerprints of duplicate blocks and their deduplication priorities into a fingerprint filter to obtain filtered fingerprints, thereby determining the actual filtering ratio. The storage system can then determine the current deduplication rate based on the actual filtering ratio. If the current deduplication rate is less than the target deduplication rate, it can adaptively adjust, for example, by increasing the deduplication ratio. The storage system then re-filters fingerprints based on the adjusted deduplication ratio until the deduplication rate reaches the target deduplication rate. This method uses adaptive adjustment to dynamically revert the deduplication rate to the achievable deduplication rate set by the system.

[0113] The storage system can then determine the blocks to be deduplicated based on the deduplication priority of the duplicate blocks and the target deduplication ratio mentioned above. For example, the storage system can sort the duplicate blocks according to the deduplication priority, and then select the blocks to be deduplicated from the sorted duplicate blocks according to the target deduplication ratio. Ultimately, this achieves the goal of selectively deduplicating some duplicate blocks within the achievable deduplication rate range, thereby reducing the number of random I / O operations and the degree of fragmentation.

[0114] S306. The storage system performs deduplication on the blocks to be deduplicated.

[0115] In some possible implementations, the storage system can directly delete the aforementioned blocks to be deleted from the persistent medium. The degree of data fragmentation in the persistent medium after the blocks to be deleted are less than the degree of data fragmentation in the persistent medium after deleting data blocks other than the blocks to be deleted from multiple duplicate blocks.

[0116] In other possible implementations, the blocks to be deduplicated include multiple blocks that meet aggregation conditions (e.g., multiple blocks identified as having locality). The storage system can also aggregate multiple blocks that meet the aggregation conditions and write them to the persistent medium, and then perform deduplication on the multiple blocks that meet the aggregation conditions. This can further reduce read fragmentation and improve read performance.

[0117] The aggregation condition can be logical address contiguousness (e.g., the ending address of one block to be deduplicated is contiguous with the starting address of another block to be deduplicated), or the interval between logical addresses is less than a set length. Alternatively, some read requests read multiple data blocks according to a step size, in which case the aggregation condition can also be that the interval between logical addresses equals the step size. In specific implementation, the storage system can sort the multiple blocks to be deduplicated determined from multiple duplicate blocks according to their logical addresses (this process is also called reordering), and then determine multiple blocks to be deduplicated that meet the aggregation condition from the sorted blocks.

[0118] The above explanation of the deduplication process is from the perspective of a single node. The following explanation is from the perspective of cross-nodes.

[0119] The storage system can add blocks to be deduplicated to a block cache for subsequent data deduplication. Specifically, any node in the storage system (e.g., a storage node) can add blocks to be deduplicated belonging to that node to the block cache. This node can also receive blocks to be deduplicated from other nodes and add them to its block cache. When the block cache meets a trigger condition, such as the number of blocks in the cache reaching a first threshold or the cache time reaching a second threshold, the storage system can sort multiple blocks to be deduplicated in the block cache according to their logical addresses. For the sorted blocks to be deduplicated, the storage system can aggregate blocks whose logical addresses meet an aggregation condition, and then perform a deduplication write, writing the aggregated blocks to the persistent storage medium.

[0120] In some possible implementations, the logical address of a block to be deduplicated in the block cache that satisfies the above aggregation condition indicates that the block to be deduplicated exhibits locality. For example... Figure 6 As shown, the storage system can first identify the locality of the blocks to be deleted, for example, by determining whether the blocks possess locality based on their logical addresses. Then, the storage system can sort the blocks that possess locality. The storage system can then aggregate these blocks, for example, by grouping them into a single unit, and simultaneously perform deduplication. Finally, the goal is to achieve contiguous storage of these local blocks on persistent media (physical space), reducing the number of random I / O operations during reads and lowering fragmentation.

[0121] To avoid storing numerous copies on persistent media, the storage system can perform deduplication on multiple deduplication blocks that meet the aggregation criteria (e.g., blocks with locality) after the aggregated block has been successfully written. Furthermore, the storage system can also deduplicatize duplicate blocks with the same fingerprint as the deduplication blocks used for aggregation after the aggregated block has been successfully written.

[0122] The following examples illustrate this. Figure 7As shown, the client generates a data stream (e.g., write IO). The data in the data stream is divided into blocks and routed to different nodes. For example, two data blocks with fingerprint FP1 (denoted as 1) are routed to node 1 and node 2 for persistence, two data blocks with fingerprint FP2 (denoted as 2) are routed to node 3 for persistence, and two data blocks with fingerprint FP6 (denoted as 6) are routed to node 1 and node 2 for persistence respectively. The two data blocks with fingerprint FP1 are duplicate blocks, as are the two data blocks with fingerprint FP2, FP3, and FP6. The storage system identifies one data block each with fingerprints FP1, FP2, FP3, and FP6 as blocks to be deleted. It further recognizes that the blocks with fingerprints FP1 and FP3 exhibit locality of reference, and the blocks with fingerprints FP2 and FP6 also exhibit locality of reference. Therefore, the blocks with fingerprints FP1 and FP3 can be aggregated and written to node 1, and the blocks with fingerprints FP2 and FP6 can be aggregated and written to node 2. The storage system then performs deduplication on these blocks, specifically deleting them from the persistent storage medium. Furthermore, the storage system can also delete duplicate blocks with the same fingerprint.

[0123] Based on the above description, it can be seen that the data deduplication method of this application embodiment can identify the differences in deduplication opportunities among different data blocks, perceive the deduplication priority of data blocks, and select blocks to be deduplicated from multiple duplicate blocks based on the deduplication priority, thereby reducing the degree of read fragmentation when reading data subsequently and improving read performance. Furthermore, this method also supports the aggregation of blocks to be deduplicated that have locality (multiple blocks to be deduplicated that meet the aggregation conditions) and batch writing them to the persistent medium (this process is also called local aggregation reorganization), thereby further reducing the degree of data fragmentation in the persistent medium and improving read performance.

[0124] The aforementioned data deduplication method can be applied to data writing scenarios. For example, after writing data blocks to persistent media, the storage system executes the aforementioned data deduplication method. This application supports the introduction of caching (e.g., online caching) to filter deduplicated data, avoiding the problem of background deduplication read / write amplification causing backpressure on foreground read / write performance due to deduplicated data being written to disk, thus improving resource utilization. Furthermore, considering that traditional caching algorithms are not designed for deduplication metadata caching and have low hit rates, some duplicate blocks cannot be identified on the write path due to low hit rates, increasing the amount of data that needs to be processed for background deduplication. Based on this, this application also designs a high-hit-rate caching algorithm to further improve caching efficiency, especially suitable for background deduplication scenarios on main memory. The following describes the data writing process of this application with reference to embodiments.

[0125] See Figure 8 The flowchart shown illustrates a data processing method, which includes:

[0126] S802, The storage system receives a write request.

[0127] Storage systems can serve as the underlying infrastructure, interfacing with upper-layer applications. Upper-layer applications (such as clients) can generate write requests; for example, a client can generate a write request in response to a user's write operation. This write request includes the data to be written. This data can typically be in stream format; in other words, the data in the write request can be a data stream.

[0128] S804, The storage system divides the data in the write request into blocks and determines the fingerprint of the data block.

[0129] S806: The storage system queries the cache based on the fingerprint of the data block. If the fingerprint is not found in the cache, execute S808. If the fingerprint is found in the cache, execute S810.

[0130] When deduplication is enabled in a storage system, the system can divide the data in a write request into blocks, resulting in at least one data block. The storage system can divide the data in the write request into blocks according to the configured deduplication granularity, resulting in at least one data block. The deduplication granularity can be configured based on empirical values; in some examples, the deduplication granularity can be configured to 4 kilobytes (KB). For each data block in the at least one data block, the storage system can determine the data block's fingerprint using a hash algorithm.

[0131] In some possible implementations, the cache can store fingerprints of deduplicated blocks. The storage system can query the cache based on the fingerprint of the data block (the fingerprint of the data block obtained after data is divided into blocks), and determine whether there is a copy of the data block in the persistent medium based on the fingerprint's hit status in the cache, thereby determining how to process the data block.

[0132] It should be noted that the data deduplication method of this application can also omit the above-described S804 and S806. For example, the storage system can also directly write the data to the persistent medium without performing data segmentation, fingerprint determination, and fingerprint comparison operations.

[0133] S808, the storage system writes data blocks to persistent media.

[0134] S810 The storage system determines the frequency of data blocks based on the reuse frequency or reuse distance of fingerprints.

[0135] S812. The storage system determines the target caching algorithm to be used for data blocks based on the frequency of the data blocks.

[0136] S814. The storage system caches the data block according to the target caching algorithm.

[0137] When a fingerprint is not found in the cache, it means that the storage system has not deduplicated the data block with the same fingerprint. The persistent medium may not contain a copy of the data block, or the number of copies of the data block is small (e.g., 1). The storage system can then write the data block to the persistent medium.

[0138] When a fingerprint is matched in the cache, it means that the storage system has previously deduplicated a data block with the same fingerprint. Since a copy of this data block exists in the persistent medium, the storage system no longer needs to perform persistence (also known as disk write-to-disk) operations. This reduces the pressure of background deduplication and mitigates the impact of background deduplication on foreground read and write operations.

[0139] For ease of description, this application may also refer to the data block in which the fingerprint is hit in the cache as the target data block. For the target data block, the storage system can determine the target caching algorithm to be used, so as to cache the target data block based on the target caching algorithm.

[0140] Specifically, the storage system can determine the frequency of the target data block (e.g., whether it is globally frequent or locally frequent), and then determine the caching strategy for the target data block based on its frequency to improve the subsequent hit rate of the data cache.

[0141] The storage system can determine the frequency of a target data block based on the fingerprint's reuse frequency or reuse distance. The fingerprint's reuse frequency or reuse distance is the fingerprint metadata. The storage system can update this fingerprint metadata when writing data blocks to persistent media. When a data block's fingerprint is not matched in the fingerprint metadata, the storage system can add the data block's fingerprint to the fingerprint metadata, setting the reuse frequency of that fingerprint to 0 and the reuse distance to ∞. When a data block's fingerprint is matched in the fingerprint metadata, the storage system can increment the reuse frequency of that fingerprint by 1 to determine the interval between that fingerprint and the last time the same fingerprint was written. For example, if the storage system writes data block 11 with fingerprint FP3, and the fingerprints of the previously written data blocks 1 to 10 are FP1, FP5, FP6, FP2, FP7, FP3, FP2, FP1, FP4, and FP8 respectively, the storage system can update the reuse frequency of FP3 to 1 and the reuse distance to 5 (e.g., the number of data blocks between them) in the fingerprint metadata.

[0142] See Figure 9Global frequency refers to the reuse frequency (number of reuses) reaching a set number within a first time window, or the reuse distance being less than a set distance. The opposite of global frequency is local frequency, which refers to the reuse frequency reaching a set number within a second time window, or the reuse distance being less than a set distance. The length of the first time window is greater than the length of the second time window; the first time window can be a long window, and the second time window can be a short window. For example, if a data block is reused 10 times within a day, it is considered a globally frequent block; if it is reused 10 times within 10 minutes, it is considered a locally frequent block. Based on this, the storage system can compare the fingerprint's reuse frequency within a time window with a set number, or compare the fingerprint's reuse distance within a time window with a set distance, to determine the frequency of a target data block. This frequency includes either local or global frequency.

[0143] In general, storage systems can choose frequency-based caching algorithms, such as Least Frequently Used (LFU), to achieve a high hit rate for globally frequent blocks. Storage systems can also determine the target caching algorithm for a target data block based on its frequency. For example, when a locally frequent block is detected, the storage system can promptly switch the caching algorithm to a recency-based algorithm, such as Least Recently Used (LRU), to achieve a high hit rate for locally frequent blocks.

[0144] If a target data block, after entering the data cache, is expected to be hit in the cache within a certain period (usually a short period), then that target data block exhibits the characteristic of local frequency. The storage system can count the number of target data blocks with this characteristic (i.e., locally frequent blocks) to determine whether a recency-based caching algorithm is necessary. Figure 10Example Explanation. In this example, the storage system can obtain the reuse distance and reuse frequency from the fingerprint metadata. Then, it arbitrates the data block based on the fingerprint of the target data block. Specifically, it obtains the reuse distance or reuse frequency of the fingerprint from the fingerprint metadata, compares the fingerprint's reuse distance with a set distance, or compares the fingerprint's reuse count with a set number of times, to check whether the target data block has a potential hit rate in the future. If the reuse distance is less than the set distance, or the reuse count is greater than the set number of times, it indicates that the target data block has local frequent characteristics and belongs to the category of locally frequent blocks. The storage system can vote on the Recency-based algorithm for this target data block (more locally frequent blocks result in greater voting weight); otherwise, it votes on the Frequency-based algorithm. Finally, the caching algorithm with the higher number of votes is selected to improve the data cache hit rate in deduplication scenarios. The storage system can use a normalized eviction algorithm, achieving the effect of LRU or LFU through different cache insertion and update rules, thereby realizing a high-hit-rate caching algorithm for main memory deduplication scenarios.

[0145] It should be noted that S810 to S814 are optional steps in the embodiments of this application. The data deduplication method of this application may also omit the above S810 to S814.

[0146] The method in this application embodiment can also be combined with the method in the foregoing embodiments. For example, after writing the data block to the persistent medium, it can be executed... Figure 3 The data deduplication method shown can meet different business needs.

[0147] As described above, this method can filter out deletable blocks through caching when writing data blocks, preventing deletable data from being written to disk, ensuring foreground read / write performance, and reducing background deleting pressure. Furthermore, this application also designs a high-hit-rate caching mechanism to further improve caching efficiency.

[0148] Based on the foregoing method embodiments, this application also provides a data deduplication apparatus. The data deduplication apparatus of this application will be described below from the perspective of functional modularity.

[0149] See Figure 11 The diagram shown illustrates the structure of a data deduplication device 1100, which includes:

[0150] The deduplication block selection module 1102 is used to respond to a data deduplication request, determine multiple duplicate blocks from data blocks stored in persistent media, and determine a block to be deduplicated from the multiple duplicate blocks according to the deduplication priority of the multiple duplicate blocks, wherein the deduplication priority of the duplicate blocks is determined based on the attributes of the duplicate blocks.

[0151] The deduplication execution module 1104 is used to perform deduplication on the block to be deduplicated.

[0152] For a detailed implementation of the deduplication block selection module 1102, please refer to [link to relevant documentation]. Figure 3 The embodiments shown describe the relevant content of S302 to S304. The deduplication execution module 1104, also known as the promote module, can be found in the following descriptions for its specific implementation: Figure 3 Description of relevant content in the illustrated embodiment S306.

[0153] In some possible implementations, the deduplication block includes multiple deduplication blocks that satisfy the aggregation condition, and the deduplication execution module 1104 is specifically used for:

[0154] The multiple deduplication blocks that meet the aggregation conditions are aggregated and then written to the persistent medium;

[0155] The multiple blocks that meet the aggregation conditions are then deduplicated.

[0156] The specific implementation of the deduplication execution module 1104 for aggregation writing and deduplication can be found in [link to relevant documentation]. Figure 3 The relevant content of embodiment S306 shown will not be repeated here.

[0157] In some possible implementations, the deduplication execution module 1104 is specifically used to determine multiple blocks to be deduplicated that satisfy the aggregation conditions in the following manner:

[0158] The multiple blocks to be deleted, determined from the multiple duplicate blocks, are sorted according to their logical addresses;

[0159] From the sorted multiple deduplication blocks, determine the multiple deduplication blocks that meet the aggregation conditions.

[0160] The specific implementation of the reordering in the deduplication execution module 1104 can be found in [link to relevant documentation]. Figure 3 The relevant content of embodiment S306 shown will not be repeated here.

[0161] In some possible implementations, the attributes of the repeating block include at least one of the following: the number of blocks routed to the same node as the repeating block, and the expected read granularity of the repeating block.

[0162] In some possible implementations, the deduplication priority of duplicate blocks is determined as follows:

[0163] When the expected read granularity of the repeated block is less than a preset value, the priority of the repeated block is determined to be the first priority.

[0164] When the expected read granularity of the duplicate block is greater than or equal to the preset value, the priority of the duplicate block is determined as the second priority based on the number of blocks routed to the same node as the duplicate block;

[0165] The first priority is higher than the second priority.

[0166] It should be noted that the process of determining the deduplication priority can be performed by the deduplication block selection module 1102. For example, the deduplication block selection module 1102 can determine the deduplication priority of the duplicate block in real time based on the attributes of the duplicate block. In some examples, the deduplication priority can also be determined when the data block identified as a duplicate block is written to the persistent medium. Based on this, the data deduplication device 1100 may also include a writing module 1101. When the writing module 1101 writes a data block to the persistent medium, it determines the deduplication priority of the data block for use in subsequent deduplication.

[0167] In some possible implementations, the repeating block includes a first repeating block and a second repeating block. When the number of blocks routed to the same node as the first repeating block is greater than the number of blocks routed to the same node as the second repeating block, the priority of the first repeating block is higher than the priority of the second repeating block.

[0168] In some possible implementations, the deduplication block selection module 1104 is also used for:

[0169] Obtain the target deduplication rate;

[0170] Pre-deletion is performed on the duplicate blocks to determine the deletion rate when the duplicate blocks are deleted according to the candidate deletion ratio;

[0171] When the deduplication rate of the duplicate block is less than the target deduplication rate when it is deduplicated according to the candidate deduplication ratio, the candidate deduplication ratio is adjusted until a target deduplication ratio is determined that makes the deduplication rate reach the target deduplication rate.

[0172] Accordingly, the deduplication block selection module 1104 is specifically used for:

[0173] Based on the deduplication priority of the multiple duplicate blocks and the target deduplication ratio, the blocks to be deduplicated are determined from the multiple duplicate blocks.

[0174] The specific implementation of the deduplication block selection module 1104, which determines the target deduplication ratio and selects the blocks to be deduplicated based on the deduplication priority and the target deduplication ratio, can be found in [link to relevant documentation]. Figure 3 Description of relevant content in the illustrated embodiment S304.

[0175] In some possible implementations, the data deduplication device 1100 may further include:

[0176] The writing module 1101 is used to receive a write request, query the cache according to the fingerprint of the data block in the write request, the cache stores the fingerprint of the deletable block, and when the fingerprint of the data block is not found in the cache, the data block is written to the persistent medium.

[0177] For a detailed implementation of the write module 1101, please refer to [link / reference]. Figure 8 The descriptions of the relevant content in the illustrated embodiments, such as the descriptions of S802 to S808, will not be repeated here.

[0178] In some possible implementations, the writing module 1101 is also used for;

[0179] When the write request includes a target data block whose fingerprint is hit in the cache, the frequency of the target data block is determined based on the reuse frequency of the fingerprint or the reuse distance. The frequency includes local frequency or global frequency. Based on the frequency of the target data block, the target caching algorithm to be used for the target data block is determined, and the target data block is cached according to the target caching algorithm.

[0180] For a detailed implementation of the write module 1101, please refer to [link / reference]. Figure 8 The descriptions of the relevant content in the illustrated embodiments, such as the descriptions of S810 to S814, will not be repeated here.

[0181] The aforementioned write module 1101 can also form an independent data processing device for writing data. For example, the data processing device may include an interaction unit, a query unit, and a persistence unit. The interaction unit receives write requests, the query unit queries a cache based on the fingerprint of the data block in the write request (the cache stores fingerprints of deleted blocks), and the persistence unit writes the data block to a persistent medium when the fingerprint of the data block is not found in the cache. Further, the data processing device may also include a caching unit, used to determine the frequency of a target data block when the write request includes a fingerprint that matches in the cache, based on the reuse frequency of the fingerprint or the reuse distance (the frequency may include local or global frequency), determine a target caching algorithm for the target data block based on the frequency, and cache the target data block according to the target caching algorithm.

[0182] It should be noted that, Figure 11The storage system is used as an example of a distributed system. The data deduplication device 1100 is deployed on each node of the distributed system. For any node's data deduplication device 1100, the deduplication block selection module 1104 can read the block to be deduplicated to the node to which the block to be deduplicated belongs (such as other nodes). Correspondingly, the deduplication execution module 1104 can receive the block to be deduplicated sent by other nodes to perform deduplication.

[0183] To address the "read fragmentation" problem caused by block-based distributed deduplication, this application designs a system aimed at reducing read amplification. By leveraging the correlation between data blocks and identifying the differences between deduplication opportunities, a natively low-fragmentation data deduplication device is provided during the data deduplication identification and execution process through priority identification and local aggregation and reorganization, thereby improving read performance. Furthermore, this device utilizes a high-hit-rate caching algorithm to cache data blocks, reducing the pressure on the background deduplication process.

[0184] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the storage system to perform the aforementioned data deduplication method.

[0185] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a storage system or stored on any usable medium. When the computer program product runs on the storage system, it causes the storage system to perform the aforementioned data deduplication method.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data deduplication method, characterized in that, The method includes: Receive a data deduplication request, the data deduplication request being used to request the deletion of duplicate data blocks in the persistent media storage; In response to the data deduplication request, multiple duplicate blocks are identified from the data block, each duplicate block having a deduplication priority, the deduplication priority of the duplicate block being determined based on the attributes of the duplicate block; Based on the deduplication priority of each duplicate block, determine the block to be deduplicated from the plurality of duplicate blocks; A deduplication operation is performed on the block to be deduplicated. After the block to be deduplicated is deleted, the fragmentation degree of the data in the persistent medium is less than a preset degree. The preset degree is the fragmentation degree of the data in the persistent medium after the data blocks other than the block to be deduplicated among the multiple duplicate blocks are deduplicated.

2. The method according to claim 1, characterized in that, The blocks to be deduplicated include multiple blocks that meet the aggregation conditions. The deduplication operation on the blocks to be deduplicated includes: The multiple deduplication blocks that meet the aggregation conditions are aggregated and then written to the persistent medium; Perform deduplication operations on the multiple blocks that meet the aggregation conditions.

3. The method according to claim 2, characterized in that, The multiple deduplication blocks that meet the aggregation conditions are determined in the following way: The multiple blocks to be deleted, determined from the multiple duplicate blocks, are sorted according to their logical addresses; From the sorted multiple deduplication blocks, determine the multiple deduplication blocks that meet the aggregation conditions.

4. The method according to claim 1, characterized in that, The attributes of the duplicate block include at least one of the following: the number of blocks routed to the same node as the duplicate block, and the expected read granularity of the duplicate block.

5. The method according to claim 4, characterized in that, The deduplication priority of the duplicate blocks is determined in the following way: When the expected read granularity of the duplicate block is less than a preset value, the deduplication priority of the duplicate block is determined to be the first priority; When the expected read granularity of the duplicate block is greater than or equal to the preset value, the deduplication priority of the duplicate block is determined to be the second priority based on the number of blocks routed to the same node as the duplicate block. The first priority is higher than the second priority.

6. The method according to claim 5, characterized in that, The duplicate blocks include a first duplicate block and a second duplicate block. When the number of blocks that are routed to the same node as the first duplicate block is greater than the number of blocks that are routed to the same node as the second duplicate block, the deduplication priority of the first duplicate block is higher than that of the second duplicate block.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Obtain the target deduplication rate; Pre-deletion is performed on the duplicate blocks to determine the deletion rate when the duplicate blocks are deleted according to the candidate deletion ratio; When the deduplication rate of the duplicate block is less than the target deduplication rate when it is deduplicated according to the candidate deduplication ratio, the candidate deduplication ratio is adjusted until a target deduplication ratio is determined that makes the deduplication rate reach the target deduplication rate. The step of determining the block to be deleted from the plurality of duplicate blocks according to the deduplication priority of each duplicate block includes: Based on the deduplication priority of each duplicate block and the target deduplication ratio, the block to be deduplicated is determined from the plurality of duplicate blocks.

8. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Receive write requests; The cache is queried based on the fingerprint of the data block in the write request, and the cache stores the fingerprint of the deleted block; If the fingerprint of the data block is not found in the cache, the data block is written to the persistent medium.

9. The method according to claim 8, characterized in that, When the write request includes a target data block whose fingerprint is hit in the cache, the method further includes; The frequency of the target data block is determined based on the reuse frequency or reuse distance of the fingerprint, and the frequency includes local frequency or global frequency. Based on the frequency of the target data block, a target caching algorithm is determined for the target data block, and the target data block is cached according to the target caching algorithm.

10. A data deduplication device, characterized in that, The device includes: The receiving module is used to receive data deduplication requests, wherein the data deduplication requests are used to request the deletion of duplicate data blocks in the data blocks stored in the persistent media; The deduplication block selection module is used to respond to the data deduplication request, determine multiple duplicate blocks from the data blocks, each duplicate block has a deduplication priority, the deduplication priority of the duplicate block is determined based on the attributes of the duplicate block; and determine the block to be deduplicated from the multiple duplicate blocks according to the deduplication priority of each duplicate block. The deduplication execution module is used to perform deduplication operations on the block to be deduplicated. After the block to be deduplicated is deleted, the fragmentation degree of the data in the persistent medium is less than a preset degree. The preset degree is the fragmentation degree of the data in the persistent medium after the data blocks other than the block to be deduplicated among the multiple duplicate blocks are deduplicated.

11. The apparatus according to claim 10, characterized in that, The deduplication block to be deleted includes multiple deduplication blocks that meet the aggregation conditions, and the deduplication execution module is specifically used for: The multiple deduplication blocks that meet the aggregation conditions are aggregated and then written to the persistent medium; Perform deduplication operations on the multiple blocks that meet the aggregation conditions.

12. The apparatus according to claim 11, characterized in that, The deduplication execution module is specifically used to determine multiple blocks to be deduplicated that meet the aggregation conditions in the following manner: The multiple blocks to be deleted, determined from the multiple duplicate blocks, are sorted according to their logical addresses; From the sorted multiple deduplication blocks, determine the multiple deduplication blocks that meet the aggregation conditions.

13. The apparatus according to claim 10, characterized in that, The attributes of the duplicate block include at least one of the following: the number of blocks routed to the same node as the duplicate block, and the expected read granularity of the duplicate block.

14. The apparatus according to claim 13, characterized in that, The deduplication priority of the duplicate blocks is determined in the following way: When the expected read granularity of the duplicate block is less than a preset value, the deduplication priority of the duplicate block is determined to be the first priority; When the expected read granularity of the duplicate block is greater than or equal to the preset value, the deduplication priority of the duplicate block is determined to be the second priority based on the number of blocks routed to the same node as the duplicate block. The first priority is higher than the second priority.

15. The apparatus according to claim 14, characterized in that, The duplicate blocks include a first duplicate block and a second duplicate block. When the number of blocks that are routed to the same node as the first duplicate block is greater than the number of blocks that are routed to the same node as the second duplicate block, the deduplication priority of the first duplicate block is higher than that of the second duplicate block.

16. The apparatus according to any one of claims 10 to 15, characterized in that, The deduplication block selection module is also used for: Obtain the target deduplication rate; Pre-deletion is performed on the duplicate blocks to determine the deletion rate when the duplicate blocks are deleted according to the candidate deletion ratio; When the deduplication rate of the duplicate block is less than the target deduplication rate when it is deduplicated according to the candidate deduplication ratio, the candidate deduplication ratio is adjusted until a target deduplication ratio is determined that makes the deduplication rate reach the target deduplication rate. The deduplication block selection module is specifically used for: Based on the deduplication priority of each duplicate block and the target deduplication ratio, the block to be deduplicated is determined from the plurality of duplicate blocks.

17. The apparatus according to any one of claims 10 to 15, characterized in that, The device further includes: The write module is used to receive write requests, query the cache based on the fingerprint of the data block in the write request, the cache stores the fingerprints of deletable blocks, and when the fingerprint of the data block is not found in the cache, the data block is written to the persistent medium.

18. The apparatus according to claim 17, characterized in that, The writing module is also used for: When the write request includes a target data block whose fingerprint is hit in the cache, the frequency of the target data block is determined based on the reuse frequency or reuse distance of the fingerprint. The frequency includes local frequency or global frequency. Based on the frequency of the target data block, the target caching algorithm to be used for the target data block is determined, and the target data block is cached according to the target caching algorithm.

19. A storage system, characterized in that, The storage system includes a controller and a memory, the memory storing computer-readable instructions; the controller executes the computer-readable instructions to cause the storage system to perform the method as described in any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images