Data deduplication method and related system

Abstract

The invention provides a data deduplication method, which comprises the following steps: in response to a data deduplication request, determining a plurality of duplicate blocks from data blocks stored in a persistent medium, and determining a to-be-deduplicated block from the plurality of duplicate blocks according to the deduplication priority of the plurality of duplicate blocks, the deduplication priority of the duplicate blocks being determined based on the attribute of the duplicate blocks, and the to-be-deduplicated block being determined based on the attribute of the duplicate blocks. And then performing deduplication on the to-be-deduplicated block, wherein the fragmentation degree of data in the persistent medium after the to-be-deduplicated block is deduplicated is smaller than the fragmentation degree of data in the persistent medium after data blocks except the to-be-deduplicated block in the plurality of duplicate blocks are deduplicated. According to the method, the deduplication priorities are set for the data blocks by analyzing different fragmentation degrees after the data blocks are deduplicated, and the blocks to be deduplicated are selected from the multiple duplicated blocks for deduplication according to the deduplication priorities, so that the fragmentation degree of reading is reduced, and the reading efficiency is improved.

Description

[0001] This application is a divisional application. The application number of the original application is 202311077281.6, and the original application date is August 24, 2023. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of storage technology, and in particular to a data deduplication method, device, storage system, computer-readable storage medium, and computer program product. Background Art

[0003] With the advent of the information age, data is growing explosively. In order to fully utilize data and mine its value, it is usually necessary to use a storage system to store data. In order to meet the growing demand for data storage, a large-capacity storage system is needed to store data, and the storage cost remains high.

[0004] Considering that the data stored in the storage system contains a large amount of duplicate data, for example, 50% of the data in the main memory system is duplicated, data deduplication technology can be used to reduce the amount of stored data, thereby reducing data storage costs. The process of data deduplication can include dividing the data into blocks, then calculating the fingerprint (FP) of the data block according to the content of the data block, identifying the duplicate block based on the fingerprint of the data block, and deleting the duplicate block, so as to achieve the purpose of reducing storage costs.

[0005] However, data deduplication can cause the originally continuously stored data to be dispersed in different areas of the storage system, such as dispersed storage in different nodes of a distributed storage system. When reading data, a continuous read input output (IO) can be degraded into multiple random read IOs. In other words, data deduplication can lead to a more serious "read fragmentation" problem, which greatly affects the read performance. Summary of the invention

[0006] The present application provides a data deduplication method, which introduces a deduplication priority and selects a to-be-deduplicated block from multiple duplicate blocks for deduplication according to the deduplication priority, thereby reducing the degree of read fragmentation and improving read efficiency. The present application also provides a device, a storage system, a computer-readable storage medium, and a computer program product corresponding to the above method.

[0007] In a first aspect, the present application provides a data deduplication method. The method may be performed by a storage system. The storage system may be a centralized storage system, including a centralized storage system with integrated disk and controller, or a centralized storage system with separate disk and controller, or the storage system may be a distributed storage system, including but not limited to a distributed storage system with integrated storage and computing, or a distributed storage system with separate storage and computing.

[0008] Specifically, in response to a data deduplication request, the storage system determines a plurality of duplicate blocks from data blocks stored in a persistent medium, determines a block to be deduplicated from the plurality of duplicate blocks according to deduplication priorities of the plurality of duplicate blocks, wherein the deduplication priorities of the duplicate blocks are determined based on attributes of the duplicate blocks, and then the storage system performs deduplication on the block to be deduplicated. The degree of data fragmentation in the persistent medium after the block to be deduplicated is deduplicated is less than the degree of data fragmentation in the persistent medium after data blocks other than the block to be deduplicated among the plurality of duplicate blocks are deduplicated.

[0009] In this method, the storage can identify the differences in deduplication opportunities of different data blocks, perceive the deduplication priority of the data blocks, and select the blocks to be deduplicated from multiple duplicate blocks based on the deduplication priority. This can achieve targeted deduplication, reduce the degree of read fragmentation when reading data subsequently, and improve read performance.

[0010] In some possible implementations, the blocks to be deduplicated include multiple blocks to be deduplicated that meet aggregation conditions. The aggregation conditions may be that the logical addresses are continuous, the interval between logical addresses is less than a set length, or the interval between logical addresses is equal to the step size. The blocks to be deduplicated that meet the above aggregation conditions may be considered to have locality (in other words, there is a high probability that the data block will be read together with other data blocks when it is subsequently read). Based on this, when the storage system performs deduplication on the above blocks to be deduplicated, it may aggregate multiple blocks to be deduplicated that meet the aggregation conditions and write them to the persistent medium, and then perform deduplication on the above multiple blocks to be deduplicated that meet the aggregation conditions.

[0011] This method also supports aggregating local blocks to be deleted (multiple blocks to be deleted that meet the aggregation conditions) and writing them to persistent media in batches (this process is also called local aggregation reorganization), thereby further reducing the degree of data fragmentation in the persistent media and improving read performance.

[0012] In some possible implementations, multiple blocks to be deleted that meet the aggregation condition are determined in the following manner: the multiple blocks to be deleted that are determined from the multiple duplicate blocks are sorted according to logical addresses, and the multiple blocks to be deleted that meet the aggregation condition are determined from the sorted multiple blocks to be deleted.

[0013] The method sorts the blocks to be deleted according to logical addresses, identifies local blocks to be deleted, aggregates the blocks to be deleted and writes them to a persistent medium, so that the local blocks to be deleted are continuously stored on the persistent medium (physical space), thereby reducing the number of random IO times and the degree of fragmentation during reading.

[0014] In some possible implementations, the attributes of the duplicate block include at least one of the number of blocks routed to the same node as the duplicate block and the expected reading granularity of the duplicate block. By analyzing factors that affect the degree of fragmentation when subsequently reading data, such as the number of blocks routed to the same node as the duplicate block and the expected reading granularity of the duplicate block, the method can more accurately determine the deduplication priority of the duplicate block and provide a reference for data deduplication.

[0015] In some possible implementations, the deduplication priority of the duplicate block is determined in the following manner: when the expected reading granularity of the duplicate block is less than a preset value, the priority of the duplicate block is determined to be the first priority; when the expected reading granularity of the duplicate block is greater than or equal to the preset value, the priority of the duplicate block is determined to be the second priority according to the number of blocks routed to the same node as the duplicate block. The first priority is higher than the second priority.

[0016] Among them, it is expected that deduplication of duplicate blocks with smaller read granularity will usually not cause fragmentation, or the degree of fragmentation is small. Based on this, this type of duplicate blocks can be deleted first. It is expected that deduplication of duplicate blocks with larger read granularity may cause fragmentation. Based on this, the deduplication priority can be determined in combination with the number of blocks routed to the same node as the duplicate block.

[0017] This method analyzes the impact of duplicate block deduplication on the degree of fragmentation from different dimensions, and prioritizes deduplication of duplicate blocks with smaller impact, thereby reducing the degree of fragmentation and improving read performance.

[0018] In some possible implementations, the repeated block includes a first repeated block and a second repeated block, and when the number of blocks routed to the same node as the first repeated block is greater than the number of blocks routed to the same node as the second repeated block, the priority of the first repeated block is higher than the priority of the second repeated block.

[0019] This method configures the deduplication priority to be positively correlated with the number of blocks routed to the same node, so that data blocks that are prone to read amplification can be deduplicated first to ensure subsequent read performance.

[0020] In some possible implementations, the storage system may also obtain a target deduplication rate, pre-deduplication the duplicate blocks, determine the deduplication rate of the duplicate blocks when deduplication is performed according to the candidate deduplication ratio, and when the deduplication rate of the duplicate blocks when deduplication is performed according to the candidate deduplication ratio is less than the target deduplication rate, adjust the candidate deduplication ratio until a target deduplication ratio is determined that enables the deduplication rate to reach the target deduplication ratio. Accordingly, the storage system may determine the blocks to be deduplicated from the multiple duplicate blocks according to the deduplication priorities of the multiple duplicate blocks and the target deduplication ratio.

[0021] In this way, the deduplication ratio can be adaptively adjusted to determine the target deduplication ratio that achieves the target deduplication ratio. The blocks to be deduplicated are selected for deduplication according to the target deduplication ratio, and the deduplication ratio can be dynamically returned to the achievable deduplication ratio set by the system.

[0022] In some possible implementations, before determining multiple duplicate blocks from the data blocks stored in the persistent medium, the storage system may also receive a write request and query a cache according to the fingerprint of the data block in the write request. The cache stores the fingerprint of the deduplicated block. When the fingerprint of the data block does not hit in the cache, the storage system may write the data block to the persistent medium.

[0023] This method introduces a cache mechanism. When writing data, it first queries the fingerprint of the data block in the cache to see if it hits. If it does not hit, the data block is written to the persistent medium. This can filter duplicate data and reduce the pressure of background deduplication, thereby avoiding the read and write amplification of background deduplication that will pressure the foreground read and write performance, thereby ensuring the foreground read and write performance.

[0024] In some possible implementations, when the write request includes a target data block hit by a fingerprint in the cache, the storage system may also determine the frequency of the target data block according to the reuse frequency of the fingerprint or the reuse distance. The frequency includes local frequency or global frequency. The storage system may determine the target cache algorithm used for the target data block according to the frequency of the target data block, and cache the target data block according to the target cache algorithm.

[0025] This method analyzes the frequency of data blocks based on the reuse frequency and reuse distance of data blocks, determines the appropriate cache strategy based on the frequency, improves the cache hit rate, and further improves the cache efficiency.

[0026] In a second aspect, the present application provides a data deduplication device. The device comprises:

[0027] a deduplication block selection module, configured to, in response to a data deduplication request, determine a plurality of duplicate blocks from data blocks stored in a persistent medium, and determine a block to be deduplicated from the plurality of duplicate blocks according to deduplication priorities of the plurality of duplicate blocks, wherein the deduplication priorities of the duplicate blocks are determined based on attributes of the duplicate blocks;

[0028] The deduplication execution module is used to perform deduplication on the block to be deduplicated, and the degree of fragmentation of the data in the persistent medium after the block to be deduplicated is less than the degree of fragmentation of the data in the persistent medium after data blocks other than the block to be deduplicated in the multiple duplicate blocks are deduplicated.

[0029] In some possible implementations, the to-be-deduplicated blocks include a plurality of to-be-deduplicated blocks that meet an aggregation condition, and the deduplication execution module is specifically configured to:

[0030] aggregating the multiple blocks to be deduplicated that meet the aggregation condition and writing them into the persistent medium;

[0031] Deduplication is performed on the multiple to-be-deduplicated blocks that meet the aggregation condition.

[0032] In some possible implementations, the deduplication execution module is specifically configured to determine a plurality of to-be-deduplicated blocks that meet the aggregation condition in the following manner:

[0033] sorting the multiple to-be-deduplicated blocks determined from the multiple duplicate blocks according to the logical addresses;

[0034] From the sorted multiple to-be-deduplicated blocks, determine the multiple to-be-deduplicated blocks that meet the aggregation condition.

[0035] In some possible implementations, the attribute of the duplicate block includes at least one of the number of blocks routed to the same node as the duplicate block and the expected reading granularity of the duplicate block.

[0036] In some possible implementations, the deduplication priority of the duplicate block is determined in the following manner:

[0037] When the expected reading granularity of the duplicate block is less than a preset value, determining the priority of the duplicate block to be the first priority;

[0038] When the expected reading granularity of the duplicate block is greater than or equal to the preset value, determining the priority of the duplicate block as the second priority according to the number of blocks routed to the same node as the duplicate block;

[0039] The first priority is higher than the second priority.

[0040] In some possible implementations, the repeated block includes a first repeated block and a second repeated block, and when the number of blocks routed to the same node as the first repeated block is greater than the number of blocks routed to the same node as the second repeated block, the priority of the first repeated block is higher than the priority of the second repeated block.

[0041] In some possible implementations, the deduplication block selection module is further configured to:

[0042] Get the target deduplication rate.

[0043] Performing pre-deduplication on the duplicate blocks to determine a deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio;

[0044] When the deduplication rate of the duplicate blocks when being deduplicated according to the candidate deduplication rate is less than the target deduplication rate, adjusting the candidate deduplication rate until a target deduplication rate is determined that enables the deduplication rate to reach the target deduplication rate;

[0045] The deduplication block selection module is specifically used for:

[0046] Determine a block to be deduplicated from the multiple duplicate blocks according to the deduplication priorities of the multiple duplicate blocks and the target deduplication ratio.

[0047] In some possible implementations, the device further includes:

[0048] The writing module is used to receive a writing request, query a cache according to the fingerprint of the data block in the writing request, the fingerprint of the deduplicated block is stored in the cache, and when the fingerprint of the data block does not hit in the cache, write the data block to a persistent medium.

[0049] In some possible implementations, the writing module is further used to:

[0050] When the write request includes a target data block whose fingerprint hits the cache, the frequency of the target data block is determined according to the reuse frequency of the fingerprint or the reuse distance, and the frequency includes local frequency or global frequency. According to the frequency of the target data block, the target cache algorithm adopted for the target data block is determined, and the target data block is cached according to the target cache algorithm.

[0051] In a third aspect, the present application provides a storage system. The storage system includes a controller and a memory. The controller and the memory communicate with each other. The controller is used to execute instructions stored in the memory so that the storage system executes the data deduplication method described in the first aspect or any implementation of the first aspect.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a storage system to execute the data deduplication method described in the first aspect or any one of the implementations of the first aspect.

[0053] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a storage system, enables the storage system to execute the data deduplication method described in the first aspect or any one of the implementations of the first aspect.

[0054] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical method of the embodiments of the present application, the drawings required for use in the embodiments are briefly introduced below.

[0056] Figure 1AA schematic diagram of the architecture of a centralized storage system with disk and controller separation provided in an embodiment of the present application;

[0057] Figure 1B A schematic diagram of the architecture of a centralized storage system with integrated disk and controller provided in an embodiment of the present application;

[0058] Figure 2A A schematic diagram of the architecture of a distributed storage system integrating storage and computing provided in an embodiment of the present application;

[0059] Figure 2B A schematic diagram of the architecture of a distributed storage system with storage and computing separation provided in an embodiment of the present application;

[0060] Figure 3 A flowchart of a data deduplication method provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of a data block routing provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of determining a deduplication priority and adjusting a deduplication ratio to achieve a achievable deduplication rate provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of a flow chart for performing deduplication on a block to be deduplicated provided in an embodiment of the present application;

[0064] Figure 7 A distribution diagram of a block to be deduplicated before and after deduplication is performed provided in an embodiment of the present application;

[0065] Figure 8 A flowchart of a data processing method provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of global frequency and local frequency provided in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of a process flow for selecting a cache algorithm provided in an embodiment of the present application;

[0068] Fig.11 A schematic diagram of the structure of a data deduplication device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0069] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0070] First, some technical terms involved in the embodiments of the present application are introduced.

[0071] Fingerprint (FP), in the field of data deduplication technology, refers to the value obtained by hashing (encrypted or unencrypted) the data content. Fingerprint can be used to represent data content.

[0072] Data deduplication usually determines the fingerprint of the data block after data segmentation, finds duplicate blocks by comparing the fingerprint of the data block, and deletes the duplicate blocks, thereby achieving the purpose of data reduction and reducing storage costs. Data deduplication can be divided into pre-deduplication and post-deduplication according to the timing of deduplication. Post-deduplication is also called background deduplication, which means deduplication after data is persistently stored. Background deduplication has lower write latency because it does not require fingerprint comparison when writing.

[0073] The data deduplication method provided in the present application can be applied to storage systems. The above storage systems can be divided into the following categories according to logical functions: primary storage and backup storage. The primary storage serves as a storage pool or hard disk for upper-layer applications and can be accessed by upper-layer applications such as applications deployed in the form of virtual machines (VMs). The backup storage serves as a warehouse for storing images, backup disks or snapshots, which are usually not directly accessible to VMs. The primary storage and backup storage can be physically separated storage systems, or the same storage system can serve as both primary storage and backup storage. Storage systems can also be divided into file storage, database storage (including row-based databases or column-based databases), and big data storage according to application scenarios.

[0074] The above storage system (including primary storage and backup storage) can adopt local storage mode or cloud storage mode. Among them, local storage refers to the mode of storing data in one's own physical storage device, and cloud storage refers to the mode of storing data in a cloud environment to support access to data through the network. The cloud environment refers to the cluster built by the cloud service provider (also known as cloud vendor) to provide computing resources, storage resources, and network resources. The storage system supports block storage, object storage, or file storage.

[0075] The storage system may adopt a centralized storage architecture or a distributed storage architecture. In other words, the storage system may be a centralized storage system or a distributed storage system. Examples are given below to illustrate each.

[0076] A centralized storage system is a central node composed of one or more main devices. Data is stored in this central node, and all data processing services of the entire system are centrally deployed on this central node. In other words, in a centralized storage system, the terminal or client is only responsible for data entry and output, while the storage and control processing of data are completely completed by the central node. The biggest feature of a centralized system is its simple deployment structure. There is no need to consider how to deploy services on multiple nodes, nor is there any need to consider the distributed collaboration between multiple nodes.

[0077] For ease of understanding, Figure 1A A centralized storage system architecture is shown. Figure 1A In the application scenario shown, users access data through applications. Computers that run these applications are called "application servers". Application server 100 can be a physical machine or a virtual machine VM. Physical machines (physical application servers) include but are not limited to desktop computers, servers, laptops, and mobile devices. The application server accesses the storage system through the fiber optic switch 110 to access data. However, the switch 110 is only an optional device, and the application server 100 can also communicate directly with the storage system 120 through the network. Alternatively, the fiber optic switch 110 can also be replaced with an Ethernet switch, an InfiniBand (IB) switch, an RDMA (RDMA over Converged Ethernet, RoCE) switch based on converged Ethernet, etc.

[0078] Figure 1A The storage system 120 shown is a centralized storage system. The centralized storage system is characterized by having a unified entrance through which all data from external devices must pass, and this entrance is the engine 121 of the centralized storage system. The engine 121 is the core component of the centralized storage system, and many advanced functions of the storage system are implemented in it.

[0079] like Figure 1A As shown, the engine 121 has one or more controllers. Figure 1ATake the example of an engine containing two controllers. There is a mirror channel between controller 0 and controller 1. Then, when controller 0 writes a copy of data into its memory 124, it can send a copy of the data to controller 1 through the mirror channel, and controller 1 stores the copy in its local memory 124. In this way, controller 0 and controller 1 back up each other. When controller 0 fails, controller 1 can take over the business of controller 0, and when controller 1 fails, controller 0 can take over the business of controller 1, thereby avoiding the unavailability of the entire storage system 120 due to hardware failure. When 4 controllers are deployed in the engine 121, there is a mirror channel between any two controllers, so any two controllers back up each other.

[0080] The engine 121 also includes a front-end interface 125 and a back-end interface 126, wherein the front-end interface 125 is used to communicate with the application server 100, thereby providing storage services for the application server 100. The back-end interface 126 is used to communicate with the hard disk 134 to expand the capacity of the storage system. Through the back-end interface 126, the engine 121 can connect more hard disks 134, thereby forming a very large storage resource pool.

[0081] In hardware, such as Figure 1A As shown, the controller 0 includes at least a processor 123 and a memory 124. The processor 123 is a central processing unit (CPU) used to process data access requests from outside the storage system (server or other storage system), and also used to process requests generated inside the storage system. Exemplarily, when the processor 123 receives a write data request sent by the application server 100 through the front-end port 125, the data in these write data requests will be temporarily stored in the memory 124. When the total amount of data in the memory 124 reaches a certain threshold, the processor 123 sends the data stored in the memory 124 to the hard disk 134 through the back-end port for persistent storage.

[0082] The memory 124 refers to an internal memory that directly exchanges data with the processor. It can read and write data at any time and at a high speed, and serves as a temporary data storage for the operating system or other running programs. The memory includes at least two types of memories, for example, the memory can be either a random access memory or a read-only memory (ROM). For example, the random access memory is a dynamic random access memory (DRAM) or a storage class memory (SCM), and the read-only memory can be a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), etc.

[0083] The memory 124 stores software programs, and the processor 123 runs the software programs in the memory 124 to manage the hard disk. For example, the hard disk is abstracted into a storage resource pool, and then divided into LUNs for use by the server. The LUN here is actually the hard disk seen on the server. Of course, some centralized storage systems are also file servers themselves, which can provide shared file services for the server.

[0084] Controller 1 (and others Figure 1A The hardware components and software structure of the controller (not shown in the figure) are similar to those of the controller 0 and will not be repeated here.

[0085] Figure 1A The illustrated embodiment is a centralized storage system with a disk-controller separation. In the system, the engine 121 may not have a hard disk slot, and the hard disk 134 needs to be placed in the hard disk frame 130, and the back-end interface 126 communicates with the hard disk frame 130. The back-end interface 126 exists in the engine 121 in the form of an adapter card, and two or more back-end interfaces 126 can be used simultaneously on one engine 121 to connect multiple hard disk frames. Alternatively, the adapter card can also be integrated on the motherboard, and the adapter card can communicate with the processor 123 via the PCIE bus.

[0086] It should be noted that Figure 1A Only one engine 121 is shown in the figure. However, in actual applications, the storage system may include two or more engines 121, and redundancy or load balancing is performed between the multiple engines 121.

[0087] The hard disk frame 130 includes a control unit 131 and a plurality of hard disks 134. The control unit 131 may have various forms. Generally, the number of control units 131 may be one, two or more. When the hard disk frame 130 includes at least two control units 131, the hard disk 134 and the control unit 131 may have an ownership relationship. The function of the control unit 131 may be offloaded to the network card 104. In this embodiment, the hard disk frame 130 does not have a control unit 131 inside, but the network card 104 completes data reading and writing, address conversion and other computing functions. At this time, the network card 104 is an intelligent network card. It may include a CPU and memory. In some application scenarios, the network card 104 may also have a persistent memory medium, such as a persistent memory (PM), or a non-volatile random access memory (NVRAM), or a phase change memory (PCM). The CPU is used to perform operations such as address conversion and reading and writing data. The memory is used to temporarily store data to be written to the hard disk 134, or data read from the hard disk 134 to be sent to the controller.

[0088] The centralized storage system can also be a disk-control integrated architecture. In the disk-control integrated architecture, the engine 121 has a hard disk slot, and the hard disk 134 can be directly deployed in the engine 121. The back-end interface 126 is an optional configuration. When the system storage space is insufficient, more hard disks or hard disk frames can be connected through the back-end interface 126. Figure 1B As shown, the engine 121 can be connected to more hard disks 134 through the backend interface 126 .

[0089] A distributed storage system is a system that stores data in multiple independent storage nodes. The distributed storage system adopts an extensible system structure and uses multiple storage nodes to share the storage load. It not only improves the reliability, availability and access efficiency of the system, but also makes it easy to expand.

[0090] Distributed storage systems can adopt a storage-computing integrated architecture or a storage-computing separated architecture. The following describes each of them.

[0091] like Figure 2A As shown, the distributed storage system integrating storage and computing includes a storage cluster. The storage cluster includes one or more servers 110 ( Figure 2AThree servers 110 are shown in the figure, but not limited to three servers 110), and each server 110 can communicate with each other. Server 110 is a device that has both computing and storage capabilities, such as a server, a desktop computer, etc. For example, an ARM server or an X86 server can be used as the server 110 here. In terms of hardware, server 110 includes at least a processor 112, a memory 113, a network card 114, and a hard disk 105. The processor 112, the memory 113, the network card 114, and the hard disk 105 are connected through a bus. Processor 112 is a central processing unit (CPU), which is used to process data access requests from outside the server 110 (application server or other server 110), and is also used to process requests generated inside the server 110. In addition, processor 112 is also used for data calculation or processing, such as metadata management, deduplication, data compression, data verification, virtualized storage space, and address conversion.

[0092] like Figure 2B As shown in FIG. 1 , the distributed storage cluster with storage and computing separation includes a computing cluster (also called a computing node cluster) and a storage cluster (also called a storage node cluster). Each computing node 110 can communicate with each other. The computing node 110 is a computing device, such as a server, a desktop computer, or a controller of a storage array. In terms of hardware, Figure 2B As shown, the computing node 110 includes at least a processor 112, a memory 113 and a network card 114. Among them, the processor 112 is a central processing unit (CPU), which is used to process data access requests from outside the computing node 110, or requests generated inside the computing node 110. When the total amount of data in the memory 113 reaches a certain threshold, the processor 112 sends the data stored in the memory 113 to the storage node 100 for persistent storage. In addition, the processor 112 is also used for data calculation or processing, such as metadata management, deduplication, data compression, virtualized storage space, and address translation.

[0093] Any computing node 110 can access any storage node 100 in the storage node cluster through the network. The storage node cluster includes multiple storage nodes 100 ( Figure 2BThree storage nodes 100 are shown in the figure, but are not limited to three storage nodes 100). A storage node 100 includes one or more controllers 101, a network card 104 and multiple hard disks 105. The network card 104 is used to communicate with the computing node 110. The hard disk 105 is used to store data and can be a disk or other types of storage media, such as a solid-state hard disk or a shingled magnetic recording hard disk. The controller 101 is used to write data to the hard disk 105 or read data from the hard disk 105 according to the read / write data request sent by the computing node 110. In the process of reading and writing data, the controller 101 needs to convert the address carried in the read / write data request into an address that can be recognized by the hard disk. It can be seen that the controller 101 also has some simple computing functions.

[0094] It should be noted that the above is merely an example description of the storage system. In other possible implementations of the embodiments of the present application, the storage system may also be a distributed storage system or a centralized storage system of other architectures. For example, the storage system may also be a distributed storage system of a fully integrated architecture.

[0095] When persisting data, the storage system can divide the data into multiple data blocks. The multiple data blocks can be routed to the persistent media of the corresponding nodes for persistent storage according to the fingerprints. Then the storage system can identify duplicate blocks in the background. After identifying the duplicate blocks, it can perform deduplication on the duplicate blocks to reduce storage costs.

[0096] However, deduplication can cause data that could have been stored continuously to be scattered and stored in different nodes or partitions. When reading data later, it was originally possible to read the data through a single continuous read IO. After deduplication, it is often necessary to read the data through multiple random read IOs (across nodes or partitions). This causes a more serious "read fragmentation" problem.

[0097] In view of this, the present application provides a data deduplication method. In response to the problem of read fragmentation, the method sets deduplication priorities for data blocks by analyzing the different degrees of fragmentation caused by data blocks being deduplicated, and selects the blocks to be deduplicated from multiple duplicate blocks according to the deduplication priorities, thereby reducing the degree of read fragmentation and improving read efficiency. Furthermore, the method can also identify the locality of duplicate blocks (if a data block has a high probability of being read together with other data blocks when it is read subsequently, the data block can be regarded as having locality), and aggregate the local duplicate blocks through a reordering method, and persist them in batches, thereby further reducing the degree of read fragmentation and further improving read efficiency.

[0098] In order to make the technical solution of the present application clearer and easier to understand, the data deduplication method of the present application is introduced below in conjunction with an embodiment.

[0099] See also Figure 3 A flow chart of a data deduplication method is shown, the method comprising the following steps:

[0100] S302: In response to a data deduplication request, the storage system determines a plurality of duplicate blocks from data blocks stored in a persistent medium.

[0101] The data deduplication request is used to trigger data deduplication. In some possible implementations, the data deduplication request may be generated by an upper-layer application of the storage system in response to a deduplication operation triggered by a user. For example, a user may turn on data deduplication in the application's settings interface, and the application generates a data deduplication request in response to the user's operation and sends the data deduplication request to the storage system. In other possible implementations, the data deduplication request may also be automatically generated by the storage system based on pre-set deduplication trigger conditions. The deduplication trigger conditions may include reaching a set deduplication time, the time interval from the last deduplication reaching a set time length, the remaining capacity of the storage system being less than a set capacity, or the duplicate data in the storage system reaching a set ratio.

[0102] Specifically, when the storage system receives a data deduplication request, it can first identify duplicate blocks with the same content from the data blocks stored in the persistent medium, and then deduplicate the duplicate blocks. Considering that the fingerprint of a data block is calculated based on the content of the data block, if the content of the data block is the same, the fingerprint of the data block is also the same, and the storage system can identify duplicate blocks by comparing the fingerprints of the data blocks.

[0103] In some possible implementations, the storage system can determine multiple duplicate blocks online. Specifically, when writing data blocks to the persistent medium, the storage system can determine the duplicate blocks in the persistent medium based on the fingerprint of the data block written to the persistent medium this time and the fingerprint of the data block previously written to the persistent medium. Specifically, the storage system can maintain fingerprint metadata, which includes the fingerprint of the data block written to the persistent medium. The storage system can determine the duplicate blocks in the persistent medium by comparing the fingerprint of the data block written to the persistent medium this time with the fingerprint in the fingerprint metadata. For example, the fingerprint of the data block written to the persistent medium this time includes FP2, and the fingerprint metadata includes the following fingerprints: FP1, FP2, FP3, then it can be determined that the data block written to the persistent medium this time and a data block with a fingerprint of FP2 previously written are duplicate blocks.

[0104] S304: The storage system determines a block to be deduplicated from the multiple duplicate blocks according to the deduplication priorities of the multiple duplicate blocks.

[0105] The storage system can perform an online theoretical analysis of the degree of fragmentation caused by deduplication of different data blocks (such as duplicate blocks), and use this to set deduplication priorities for different data blocks (such as duplicate blocks). Among them, data blocks with high deduplication priorities are deleted before data blocks with low deduplication priorities. It should be noted that the storage system can determine the deduplication priority when a data block identified as a duplicate block is written to a persistent medium. The storage system can store the deduplication priority of the data block, so that the block to be deleted is selected based on the deduplication priority during the subsequent deduplication process. The storage system can also determine the deduplication priority of the duplicate block in real time during the deduplication process.

[0106] Among them, the deduplication priority of a data block (for example, a data block identified as a duplicate block) can be determined based on the attributes of the data block. In some possible implementations, the attributes of the data block may include at least one of the expected read granularity and the number of blocks routed to the same node as the data block. Based on this, for duplicate blocks, the storage system can determine the deduplication priority of the duplicate block based on the expected read granularity of the duplicate block and / or the number of blocks routed to the same node as the duplicate block. Among them, the number of blocks routed to the same node as the duplicate block refers to the number of blocks routed to the same node as the duplicate block in the same write request (write IO). For example, the data in a write request is divided into data blocks 1 to 5, wherein data block 1 is a duplicate block, and data blocks 1, 3, and 5 are routed to node 1, then the number of blocks routed to the same node as data block 1, which is a duplicate block, is 2.

[0107] In a specific implementation, the storage system (for example, a storage node in the storage system) can obtain the expected read granularity of the duplicate block. When the expected read granularity of the duplicate block is less than the preset value, it means that the duplicate block is a data block that will not cause read amplification after deduplication, or the duplicate block is a data block with a small degree of fragmentation after deduplication, and the storage system can determine that the priority of the duplicate block is the first priority. In some examples, the first priority can be the highest priority. When the expected read granularity of the duplicate block is greater than or equal to the preset value, it means that the duplicate block is a data block that will cause read amplification after deduplication, or the duplicate block is a data block with a large degree of fragmentation after deduplication, and the storage system can determine that the priority of the duplicate block is the second priority based on the number of blocks routed to the same node as the duplicate block. The first priority is higher than the second priority. It should be noted that the above preset value can be set based on experience, for example, it can be set to be equal to the deduplication granularity. In some examples, the preset value can be set to 4KB.

[0108] The expected read granularity refers to the expected value of the granularity when it is subsequently read. In some possible implementations, the expected read granularity may be equal to the granularity of the write request (or write IO) to which the data block belongs. For example, if the amount of data in the write request is 1 megabyte (MB), then after the data block obtained by the write request after data segmentation is written to the persistent medium, if a data block is identified as a duplicate block, the expected read granularity of the duplicate block may be 1MB. In some other possible implementations, the expected read granularity may also be determined by statistical analysis of historical data. For example, the storage system may determine the expected read granularity based on the number of times the previously written data block with the same fingerprint is accessed and the read IO size each time it is accessed. Among them, the storage system may determine the average value of the read IO size based on the number of times the data block with the same fingerprint is accessed and the read IO size each time it is accessed, and the storage system may determine the average value as the expected read granularity. Alternatively, the storage system may also determine the median value of the read IO size each time it is accessed, and the storage system may determine the median value as the expected read granularity.

[0109] Further, the second priority may be positively correlated with the number of blocks that the repeated block is routed to the same node. Specifically, the repeated block may include a first repeated block and a second repeated block, and when the number of blocks that the first repeated block is routed to the same node is greater than the number of blocks that the second repeated block is routed to the same node, the priority of the first repeated block is higher than the priority of the second repeated block.

[0110] For ease of understanding, the following is an example. Figure 4 As shown, the data stream includes two data blocks with fingerprint FP1 (recorded as data blocks 1 and 2), two data blocks with fingerprint FP2 (recorded as data blocks 3 and 5), and two data blocks with fingerprint FP3 (recorded as data blocks 4 and 6), wherein data blocks 1 and 2 and data blocks 3 and 5 are routed to node 1 (node ​​1), and data blocks 4 and 6 are routed to node 2 (node ​​2). The above data blocks 1 and 2 and data blocks 3 and 5 and data blocks 4 and 6 are identified as duplicate blocks. This example assumes that the expected read granularity of the above data blocks is greater than 4KB, and the storage system can determine the priority of duplicate blocks according to the number of blocks routed to the same node as the duplicate blocks. For data blocks 1, 2, 3, and 5, the number of blocks routed to the same node as the data blocks is 3, and for data blocks 4 and 6, the number of blocks routed to the same node as the data blocks is 1, so it can be determined that the deduplication priority of data blocks 1, 2, 3, and 5 is higher than that of data blocks 4 and 6.

[0111] When selecting the blocks to be deduplicated, the storage system may give priority to the duplicate blocks with higher deduplication priorities. In some possible implementations, the storage system may sort the duplicate blocks from high to low according to the deduplication priorities of the duplicate blocks, and then select the top n or top m% of the duplicate blocks in the sorting order to determine them as the blocks to be deduplicated. In other possible implementations, the storage system may also select duplicate blocks with a deduplication priority higher than a preset priority as the blocks to be deduplicated.

[0112] Furthermore, the storage system can also obtain a target deduplication rate, which refers to the achievable deduplication rate of the storage system. The storage system can pre-deduplication the duplicate blocks and determine the deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio. When the deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio is less than the target deduplication rate, the storage system can adjust the candidate deduplication ratio until the target deduplication ratio is determined so that the deduplication rate reaches the target deduplication rate. Figure 5 As shown, the storage system can input the fingerprint of the duplicate block and its deduplication priority into the fingerprint filter to obtain the filtered fingerprint, thereby determining the actual filtering ratio. The storage system can determine the current deduplication rate based on the actual filtering ratio. If the current deduplication rate is less than the target deduplication rate, it can be adaptively adjusted, for example, by increasing the deduplication rate, and then the storage system re-performs fingerprint filtering based on the adjusted deduplication rate until the deduplication rate reaches the target deduplication rate. This method uses adaptive adjustment to dynamically return the deduplication rate to the achievable deduplication rate set by the system.

[0113] Then the storage system can determine the blocks to be deduplicated according to the deduplication priorities of the duplicate blocks and the above target deduplication ratio. For example, the storage system can sort the duplicate blocks according to the deduplication priorities, and then select the blocks to be deduplicated from the sorted duplicate blocks according to the target deduplication ratio, and finally achieve the purpose of selectively deduplicating some duplicate blocks within the reachable deduplication ratio, thereby reducing the number of random IO times and the degree of fragmentation.

[0114] S306: The storage system performs deduplication on the block to be deduplicated.

[0115] In some possible implementations, the storage system may directly delete the block to be deduplicated from the persistent medium, wherein the degree of data fragmentation in the persistent medium after the block to be deduplicated is deduplicated is less than the degree of data fragmentation in the persistent medium after data blocks other than the block to be deduplicated in the plurality of duplicate blocks are deduplicated.

[0116] In some other possible implementations, the blocks to be deduplicated include multiple blocks to be deduplicated that meet the aggregation condition (for example, multiple blocks to be deduplicated that are identified as having locality), and the storage system may also aggregate the multiple blocks to be deduplicated that meet the aggregation condition and write them to the persistent medium, and then perform deduplication on the multiple blocks to be deduplicated that meet the aggregation condition. In this way, the degree of read fragmentation can be further reduced and the read performance can be improved.

[0117] The aggregation condition may be that the logical addresses are continuous (for example, the end address of one block to be deleted is continuous with the start address of another block to be deleted), or that the interval between logical addresses is less than a set length. Alternatively, if some read requests read multiple data blocks according to a step length, the aggregation condition may also be that the interval between logical addresses is equal to the step length. In specific implementation, the storage system may sort the multiple blocks to be deleted determined from the multiple duplicate blocks according to the logical addresses (this process is also called reordering), and then determine multiple blocks to be deleted that meet the aggregation condition from the sorted multiple blocks to be deleted.

[0118] The above describes the deduplication process from the perspective of a single node. The following describes the deduplication process from the perspective of a cross-node node.

[0119] The storage system can add the blocks to be deleted to the block cache to perform subsequent data deduplication. Among them, for any node of the storage system (for example, a storage node), the blocks to be deleted belonging to the node can be added to the block cache, and the node can also receive the blocks to be deleted sent by other nodes and add them to the block cache. When the block cache meets the trigger condition, for example, when the number of blocks in the block cache reaches the first threshold or the cache time reaches the second threshold, the storage system can sort the multiple blocks to be deleted in the block cache according to the logical address. For the sorted multiple blocks to be deleted, the storage system can aggregate the blocks to be deleted whose logical addresses meet the aggregation condition, and then perform deduplication writing, and write the aggregated blocks to the persistent medium.

[0120] In some possible implementations, if the logical address of the block to be deduplicated in the block cache satisfies the above aggregation condition, it means that the block to be deduplicated has locality. Figure 6 As shown, the storage system can first identify the locality of the blocks to be deleted, for example, determine whether the blocks to be deleted are local based on the logical addresses of the blocks to be deleted. Then the storage system can sort the local blocks to be deleted. The storage system can aggregate the local blocks to be deleted, for example, aggregate them into a whole, and perform deduplication and write at the same time. Finally, the local blocks to be deleted are continuously stored on the persistent medium (physical space), and the number of random IO times and the degree of fragmentation are reduced during reading.

[0121] In order to avoid storing too many copies in the persistent medium, the storage system can perform deduplication based on multiple to-be-deduplicated blocks that meet the aggregation conditions (for example, to-be-deduplicated blocks with locality) after the aggregated blocks are written successfully. Furthermore, the storage system can also perform deduplication on duplicate blocks with the same fingerprint as the to-be-deduplicated blocks used for aggregation after the aggregated blocks are written successfully.

[0122] The following is an example. Figure 7As shown, the client generates a data stream (for example, write IO), and the data in the data stream is divided into blocks and routed to different nodes. For example, two data blocks with fingerprint FP1 (denoted as 1) are routed to node 1 and node 2 for persistence, two data blocks with fingerprint FP2 (denoted as 2) are routed to node 3 for persistence, and two data blocks with fingerprint FP6 (denoted as 6) are routed to node 1 and node 2 for persistence respectively. The two data blocks with fingerprint FP1 are duplicate blocks, the two data blocks with fingerprint FP2 are duplicate blocks, the two data blocks with fingerprint FP3 are duplicate blocks, and the two data blocks with fingerprint FP6 are duplicate blocks. The storage system determines that one data block with fingerprint FP1, FP2, FP3, and FP6 is a block to be deleted, and recognizes that the blocks to be deleted with fingerprint FP1 and FP3 are local, and the blocks to be deleted with fingerprint FP2 and FP6 are local. Therefore, the blocks to be deleted with fingerprint FP1 and FP3 can be aggregated and written to node 1, and the blocks to be deleted with fingerprint FP2 and FP6 can be aggregated and written to node 2. Then the storage system deduplicates the blocks to be deleted, specifically, deletes them from the persistent medium. Furthermore, the storage system can also delete duplicate blocks with the same fingerprint.

[0123] Based on the above description, it can be known that the data deduplication method of the embodiment of the present application can identify the differences in deduplication opportunities of different data blocks, perceive the deduplication priority of the data block, and select the block to be deduplicated from multiple duplicate blocks based on the deduplication priority for deduplication, thereby reducing the degree of read fragmentation during subsequent data reading and improving read performance. In addition, the method also supports the aggregation of local blocks to be deduplicated (multiple blocks to be deduplicated that meet the aggregation conditions) and batch writing to persistent media (this process is also called local aggregation reorganization), thereby further reducing the degree of data fragmentation in the persistent medium and improving read performance.

[0124] The aforementioned data deduplication method can be applied to scenarios where data is written. For example, after the storage system writes a data block to a persistent medium, it executes the aforementioned data deduplication method. The present application supports the introduction of a cache (for example, an online cache) to filter the deduplicated data, thereby avoiding the problem of background deduplication read and write amplification and back pressure on foreground read and write performance caused by the deduplicated data being stored on the disk, thereby improving resource utilization. Furthermore, considering that traditional cache algorithms are not designed for deduplication metadata caching, the hit rate is low. The low hit rate causes some duplicate blocks to be unable to be identified on the write path, increasing the amount of data that needs to be processed by the background deduplication. Based on this, the present application also designs a high-hit rate cache algorithm to further improve cache efficiency, which is particularly suitable for main memory-oriented background deduplication scenarios. The data writing of the present application is introduced below in conjunction with the embodiments.

[0125] See also Figure 8 A flow chart of a data processing method is shown, the method comprising:

[0126] S802: The storage system receives a write request.

[0127] The storage system can be used as the underlying infrastructure to connect to the upper-layer application. The upper-layer application (e.g., the client) can generate a write request. For example, the client can generate a write request in response to a user's write operation. The write request includes the data to be written. The data can usually be in a stream form. In other words, the data in the write request can be a data stream.

[0128] S804: The storage system divides the data in the write request into blocks and determines fingerprints of the data blocks.

[0129] S806, the storage system queries the cache according to the fingerprint of the data block. If the fingerprint does not hit in the cache, execute S808. If the fingerprint hits in the cache, execute S810.

[0130] When the storage system turns on deduplication, the storage system may divide the data in the write request into blocks, thereby obtaining at least one data block. The storage system may divide the data in the write request into blocks according to the configured deduplication granularity, thereby obtaining at least one data block. The deduplication granularity may be configured according to an empirical value. In some examples, the deduplication granularity may be configured to 4 kilobytes (KB). For each data block in the at least one data block, the storage system may determine the fingerprint of the data block through a hash algorithm.

[0131] In some possible implementations, the cache can store fingerprints of deduplicated blocks. The storage system can query the cache based on the fingerprint of the data block (the fingerprint of the data block obtained after data segmentation), and can determine whether the data block has a copy in the persistent medium based on the hit of the fingerprint in the cache, and then determine how to handle the data block.

[0132] It should be noted that the data deduplication method of the present application may not perform the above S804 and S806. For example, the storage system may not perform data segmentation, fingerprint determination, and fingerprint comparison operations, and directly write the data to the persistent medium.

[0133] S808: The storage system writes the data block to a persistent medium.

[0134] S810: The storage system determines the frequency of the data block according to the reuse frequency or reuse distance of the fingerprint.

[0135] S812: The storage system determines a target cache algorithm to be used for the data block according to the frequency of the data block.

[0136] S814: The storage system caches the data block according to a target cache algorithm.

[0137] When the fingerprint does not hit in the cache, it means that the storage system has not deduplicated the data block with the same fingerprint, the persistent medium may not include a copy of the data block, or the number of copies of the data block is small (for example, 1), and the storage system can write the data block to the persistent medium.

[0138] When the fingerprint hits in the cache, it means that the storage system has deleted duplicate data blocks with the same fingerprint. A copy of the data block exists in the persistent medium. The storage system no longer needs to perform persistence (also known as flushing to disk or removing from disk). This can reduce the pressure of background deduplication and the impact of background deduplication on foreground reading and writing.

[0139] For ease of description, the present application may also refer to the data block hit by the fingerprint in the cache as the target data block. For the target data block, the storage system may determine the target cache algorithm used for the target data block so as to cache the target data block based on the target cache algorithm.

[0140] Specifically, the storage system may determine the frequency of the target data block (eg, global frequency or local frequency), and then determine a cache strategy for the target data block according to the frequency of the target data block to improve subsequent hit rates of the data cache.

[0141] Among them, the storage system can determine the frequency of the target data block according to the reuse frequency (or reuse frequency) or reuse distance of the fingerprint. The reuse frequency of the fingerprint) or reuse distance is the fingerprint metadata. The storage system can update the above fingerprint metadata when writing the data block to the persistent medium. When the fingerprint of the data block does not hit in the fingerprint metadata, the storage system can add the fingerprint of the data block to the fingerprint metadata, record the reuse frequency of the fingerprint as 0, and record the reuse distance as ∞. When the fingerprint of the data block hits in the fingerprint metadata, the storage system can add 1 to the reuse frequency of the fingerprint to determine the interval between the fingerprint and the last time the same fingerprint was written. For example, the storage system writes data block 11 with a fingerprint of FP3, and the fingerprints of the previously written data blocks 1 to 10 are FP1, FP5, FP6, FP2, FP7, FP3, FP2, FP1, FP4, and FP8, respectively. Based on this, the storage system can update the reuse frequency of FP3 to 1 and the reuse distance to 5 (for example, the number of interval data blocks) in the fingerprint metadata.

[0142] See also Fig. 9, global frequency means that the reuse frequency (number of reuses) reaches the set number of times in the first time window, or the reuse distance is less than the set distance; corresponding to global frequency is local frequency, which means that the reuse frequency reaches the set number of times in the second time window, or the reuse distance is less than the set distance. The length of the first time window is greater than the length of the second time window. The first time window can be a long time window, and the second time window can be a short time window. For example, if the reuse frequency reaches 10 times within 1 day, the data block is considered to be a global frequent block, and if the reuse frequency reaches 10 times within 10 minutes, the data block is considered to be a local frequent block. Based on this, the storage system can compare the reuse frequency of the fingerprint in the time window with the set number of times, or compare the reuse distance of the fingerprint in the time window content with the set distance to determine the frequency of the target data block. The frequency includes local frequency or global frequency.

[0143] In general, the storage system can select a frequency-based cache algorithm, such as the Least Frequently Used (LFU) algorithm, to achieve a high hit rate for global frequent blocks. The storage system can determine the target cache algorithm used for the target data block based on the frequency of the target data block. For example, when the storage system analyzes the arrival of a local frequent block, it can promptly switch the cache algorithm to a recency-based cache algorithm, such as the Least Recently Used (LRU) algorithm, to achieve a high hit rate for local frequent blocks.

[0144] After a target data block enters the data cache, if it will be hit in the data cache in the future (usually a short period of time), then the target data block has the characteristic of local frequency. The storage system can count the number of target data blocks with local frequency characteristics (i.e., local frequent blocks) to determine whether it is necessary to use a recency-based cache algorithm. Fig.10Example description. In this example, the storage system can obtain the reuse distance and reuse frequency from the fingerprint metadata, and then arbitrate the data block according to the fingerprint of the target data block. Specifically, the reuse distance or reuse frequency of the fingerprint is obtained from the fingerprint metadata, and the reuse distance of the fingerprint is compared with the set distance, or the reuse number of the fingerprint is compared with the set number, to check whether the target data block has a potential hit in the future. If the reuse distance is less than the set distance, or the reuse number is greater than the set number, it means that the target data block has a local frequent feature and belongs to a local frequent block. The storage system can vote for the Recency-based algorithm for the target data block (there are many local frequent blocks and the voting power is large). Otherwise, it votes for the Frequency-based algorithm. Finally, the cache algorithm with a higher number of votes is selected to improve the hit rate of the data cache in the deduplication scenario. Among them, the storage system can adopt a normalized elimination algorithm to achieve the effect of LRU or LFU through different cache insertion and update rules, thereby realizing a cache algorithm with a high hit rate in the main memory deduplication scenario.

[0145] It should be noted that the above S810 to S814 are optional steps of the embodiment of the present application, and the data deduplication method of the present application may be executed without executing the above S810 to S814.

[0146] The method of the embodiment of the present application can also be combined with the method of the above embodiment. For example, after writing the data block to the persistent medium, the following method can be executed: Figure 3 The data deduplication method shown can meet different business needs.

[0147] Based on the above description, it can be seen that this method can filter the deduplicated blocks through the cache when writing data blocks, avoid the deduplicated data from being written to the disk, ensure the front-end read and write performance, and reduce the back-end deduplication pressure. In addition, this application also designs a high hit rate cache mechanism to further improve the cache efficiency.

[0148] Based on the above method embodiments, the present application also provides a data deduplication device. The data deduplication device of the present application will be introduced from the perspective of functional modularization.

[0149] See also Fig.11 The structure diagram of a data deduplication device shown in FIG. 1 includes:

[0150] a deduplication block selection module 1102, configured to determine, in response to a data deduplication request, a plurality of duplicate blocks from data blocks stored in a persistent medium, and determine a block to be deduplicated from the plurality of duplicate blocks according to deduplication priorities of the plurality of duplicate blocks, wherein the deduplication priorities of the duplicate blocks are determined based on attributes of the duplicate blocks;

[0151] The deduplication execution module 1104 is used to perform deduplication on the blocks to be deduplicated.

[0152] The specific implementation of the deduplication block selection module 1102 can be found in Figure 3 The deduplication execution module 1104 is also called a promote module. The specific implementation of the deduplication execution module 1104 can be found in Figure 3 The relevant contents of the embodiment S306 are described.

[0153] In some possible implementations, the block to be deduplicated includes multiple blocks to be deduplicated that meet the aggregation condition, and the deduplication execution module 1104 is specifically configured to:

[0154] aggregating the multiple blocks to be deduplicated that meet the aggregation condition and writing them into the persistent medium;

[0155] Deduplication is performed on the multiple to-be-deduplicated blocks that meet the aggregation condition.

[0156] The specific implementation of the aggregate writing and deduplication by the deduplication execution module 1104 can be found in Figure 3 The description of the relevant contents of S306 of the illustrated embodiment will not be repeated here.

[0157] In some possible implementations, the deduplication execution module 1104 is specifically configured to determine a plurality of to-be-deduplicated blocks that meet the aggregation condition in the following manner:

[0158] sorting the multiple to-be-deduplicated blocks determined from the multiple duplicate blocks according to the logical addresses;

[0159] From the sorted multiple to-be-deduplicated blocks, determine the multiple to-be-deduplicated blocks that meet the aggregation condition.

[0160] The specific implementation of reordering by the deduplication execution module 1104 can be found in Figure 3 The description of the relevant contents of S306 of the illustrated embodiment will not be repeated here.

[0161] In some possible implementations, the attribute of the duplicate block includes at least one of the number of blocks routed to the same node as the duplicate block and the expected reading granularity of the duplicate block.

[0162] In some possible implementations, the deduplication priority of duplicate blocks is determined as follows:

[0163] When the expected reading granularity of the duplicate block is less than a preset value, determining the priority of the duplicate block to be the first priority;

[0164] When the expected reading granularity of the duplicate block is greater than or equal to the preset value, determining the priority of the duplicate block as the second priority according to the number of blocks routed to the same node as the duplicate block;

[0165] The first priority is higher than the second priority.

[0166] It should be noted that the process of determining the deduplication priority may be performed by the deduplication block selection module 1102. For example, the deduplication block selection module 1102 may determine the deduplication priority of the duplicate block in real time according to the attributes of the duplicate block. In some examples, the deduplication priority may also be determined when the data block identified as a duplicate block is written to a persistent medium. Based on this, the data deduplication device 1100 may also include a writing module 1101. When the writing module 1101 writes a data block to a persistent medium, it determines the deduplication priority of the data block for use in subsequent deduplication.

[0167] In some possible implementations, the repeated block includes a first repeated block and a second repeated block, and when the number of blocks routed to the same node as the first repeated block is greater than the number of blocks routed to the same node as the second repeated block, the priority of the first repeated block is higher than the priority of the second repeated block.

[0168] In some possible implementations, the deduplication block selection module 1104 is further configured to:

[0169] Get the target deduplication rate.

[0170] Performing pre-deduplication on the duplicate blocks to determine a deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio;

[0171] When the deduplication rate of the duplicate blocks when being deduplicated according to the candidate deduplication rate is less than the target deduplication rate, adjusting the candidate deduplication rate until a target deduplication rate is determined that enables the deduplication rate to reach the target deduplication rate;

[0172] Accordingly, the deduplication block selection module 1104 is specifically used for:

[0173] Determine a block to be deduplicated from the multiple duplicate blocks according to the deduplication priorities of the multiple duplicate blocks and the target deduplication ratio.

[0174] The specific implementation of the deduplication block selection module 1104 determining the target deduplication ratio and determining the to-be-deduplicated blocks according to the deduplication priority and the target deduplication ratio can be seen in Figure 3 The relevant contents of S304 of the embodiment shown are described.

[0175] In some possible implementations, the data deduplication apparatus 1100 may further include:

[0176] The write module 1101 is used to receive a write request, query a cache according to the fingerprint of the data block in the write request, the cache stores the fingerprint of the deduplicated block, and write the data block to a persistent medium when the fingerprint of the data block does not hit in the cache.

[0177] The specific implementation of the writing module 1101 can be found in Figure 8 The description of the relevant contents of the illustrated embodiment, such as the description of the relevant contents of S802 to S808, will not be repeated here.

[0178] In some possible implementations, the writing module 1101 is further used for:

[0179] When the write request includes a target data block whose fingerprint hits the cache, the frequency of the target data block is determined according to the reuse frequency of the fingerprint or the reuse distance, and the frequency includes local frequency or global frequency. According to the frequency of the target data block, the target cache algorithm adopted for the target data block is determined, and the target data block is cached according to the target cache algorithm.

[0180] The specific implementation of the writing module 1101 can be found in Figure 8 The description of the relevant contents of the illustrated embodiment, such as the description of the relevant contents of S810 to S814, will not be repeated here.

[0181] The above-mentioned writing module 1101 can also form an independent data processing device for writing data. For example, the data processing device may include an interaction unit, a query unit, and a persistence unit. The interaction unit is used to receive a write request, the query unit is used to query the cache according to the fingerprint of the data block in the write request, the cache stores the fingerprint of the deleted block, and the persistence unit is used to write the data block to a persistent medium when the fingerprint of the data block does not hit in the cache. Furthermore, the data processing device may also include a cache unit, which is used to determine the frequency of the target data block according to the reuse frequency of the fingerprint or the reuse distance when the write request includes a target data block whose fingerprint hits in the cache, the frequency includes local frequency or global frequency, and the target cache algorithm used for the target data block is determined according to the frequency of the target data block, and the target data block is cached according to the target cache algorithm.

[0182] It should be noted that Fig.11The storage system is used as an example of a distributed system. The data deduplication device 1100 is deployed in each node of the distributed system. For the data deduplication device 1100 of any node, the deduplication block selection module 1104 can read the block to be deduplicated to the node to which the block to be deduplicated belongs (such as other nodes). Correspondingly, the deduplication execution module 1104 can receive the block to be deduplicated sent by other nodes to perform deduplication.

[0183] In response to the "read fragmentation" problem caused by distributed deduplication based on block routing, this application designs the system with the goal of reducing read amplification. By utilizing the correlation between data blocks and the differences between identifying deduplication opportunities, a native low-fragmentation data deduplication device is provided in the process of data deduplication identification and execution through priority identification and local aggregation and reorganization to improve read performance. In addition, the device uses a high-hit rate cache algorithm to cache data blocks to reduce the pressure of background deduplication.

[0184] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the storage system to execute the aforementioned data deduplication method.

[0185] The embodiment of the present application also provides a computer program product including instructions. The computer program product may be a software or program product including instructions that can be run on a storage system or stored in any available medium. When the computer program product is run on a storage system, the storage system executes the above-mentioned data deduplication method.

[0186] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data deduplication method, characterized in that: The method comprises: receiving a data deduplication request, wherein the data deduplication request is used to request deletion of duplicate data blocks in data blocks stored in a persistent medium; In response to the data deduplication request, determining a plurality of duplicate blocks from the data block, each duplicate block having a deduplication priority, and the deduplication priority of the duplicate block is determined based on an attribute of the duplicate block; Determining a block to be deduplicated from the multiple duplicate blocks according to the deduplication priority of each duplicate block; A deduplication operation is performed on the block to be deduplicated, and after the block to be deduplicated is deleted, the degree of fragmentation of the data in the persistent medium is less than a preset degree, and the preset degree is the degree of fragmentation of the data in the persistent medium after data blocks other than the block to be deduplicated among the multiple duplicate blocks are deduplicated.

2. The method according to claim 1, characterized in that: The blocks to be deduplicated include a plurality of blocks to be deduplicated that meet an aggregation condition, and performing a deduplication operation on the blocks to be deduplicated includes: aggregating the multiple blocks to be deduplicated that meet the aggregation condition and writing them into the persistent medium; A deduplication operation is performed on the multiple to-be-deduplicated blocks that meet the aggregation condition.

3. The method according to claim 2, characterized in that The multiple blocks to be deduplicated that meet the aggregation condition are determined in the following manner: sorting the multiple to-be-deduplicated blocks determined from the multiple duplicate blocks according to the logical addresses; From the sorted multiple to-be-deduplicated blocks, determine the multiple to-be-deduplicated blocks that meet the aggregation condition.

4. The method according to any one of claims 1 to 3, characterized in that: The attribute of the duplicate block includes at least one of the number of blocks routed to the same node as the duplicate block and the expected read granularity of the duplicate block.

5. The method according to claim 4, characterized in that The deduplication priority of the duplicate blocks is determined in the following manner: When the expected reading granularity of the duplicate block is less than a preset value, determining the priority of the duplicate block to be the first priority; When the expected reading granularity of the duplicate block is greater than or equal to the preset value, determining the priority of the duplicate block as the second priority according to the number of blocks routed to the same node as the duplicate block; The first priority is higher than the second priority.

6. The method according to claim 5, characterized in that The repeated blocks include a first repeated block and a second repeated block. When the number of blocks routed to the same node as the first repeated block is greater than the number of blocks routed to the same node as the second repeated block, the priority of the first repeated block is higher than that of the second repeated block.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Get the target deduplication rate; Performing pre-deduplication on the duplicate blocks to determine a deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio; When the deduplication rate of the duplicate blocks when being deduplicated according to the candidate deduplication rate is less than the target deduplication rate, adjusting the candidate deduplication rate until a target deduplication rate is determined that enables the deduplication rate to reach the target deduplication rate; The step of determining a block to be deduplicated from the multiple duplicate blocks according to the deduplication priority of each duplicate block includes: A block to be deduplicated is determined from the multiple duplicate blocks according to the deduplication priority of each duplicate block and the target deduplication ratio.

8. The method according to any one of claims 1 to 7, characterized in that: Before determining a plurality of duplicate blocks from the data blocks stored in the persistent medium, the method further includes: Receive write request; querying a cache according to the fingerprint of the data block in the write request, wherein the cache stores the fingerprint of the deduplicated block; When the fingerprint of the data block does not hit in the cache, the data block is written to the persistent medium.

9. The method according to claim 8, characterized in that When the write request includes a target data block whose fingerprint hits in the cache, the method further includes: Determining the frequency of the target data block according to the reuse frequency of the fingerprint or the reuse distance, wherein the frequency includes local frequency or global frequency; According to the frequency of the target data block, a target cache algorithm adopted for the target data block is determined, and the target data block is cached according to the target cache algorithm.

10. A data deduplication device, characterized in that: The device comprises: A receiving module, used for receiving a data deduplication request, wherein the data deduplication request is used for requesting to delete duplicate data blocks in data blocks stored in a persistent medium; a deduplication block selection module, configured to, in response to the data deduplication request, determine a plurality of duplicate blocks from the data block, each duplicate block having a deduplication priority, the deduplication priority of the duplicate block being determined based on an attribute of the duplicate block; and determine a block to be deduplicated from the plurality of duplicate blocks according to the deduplication priority of each duplicate block; A deduplication execution module is used to perform a deduplication operation on the block to be deduplicated, and the degree of fragmentation of the data in the persistent medium after the block to be deduplicated is deleted is less than a preset degree, and the preset degree is the degree of fragmentation of the data in the persistent medium after data blocks other than the block to be deduplicated among the multiple duplicate blocks are deduplicated.

11. The device according to claim 10, characterized in that The to-be-deduplicated blocks include a plurality of to-be-deduplicated blocks that meet aggregation conditions, and the deduplication execution module is specifically configured to: aggregating the multiple blocks to be deduplicated that meet the aggregation condition and writing them into the persistent medium; A deduplication operation is performed on the multiple to-be-deduplicated blocks that meet the aggregation condition.

12. The device according to claim 11, characterized in that The deduplication execution module is specifically used to determine a plurality of to-be-deduplicated blocks that meet the aggregation condition in the following manner: sorting the multiple to-be-deduplicated blocks determined from the multiple duplicate blocks according to the logical addresses; From the sorted multiple to-be-deduplicated blocks, determine the multiple to-be-deduplicated blocks that meet the aggregation condition.

13. The device according to any one of claims 10 to 12, characterized in that The attribute of the duplicate block includes at least one of the number of blocks routed to the same node as the duplicate block and the expected read granularity of the duplicate block.

14. The device according to claim 13, characterized in that The deduplication priority of the duplicate blocks is determined in the following manner: When the expected reading granularity of the duplicate block is less than a preset value, determining the priority of the duplicate block to be the first priority; When the expected reading granularity of the duplicate block is greater than or equal to the preset value, determining the priority of the duplicate block as the second priority according to the number of blocks routed to the same node as the duplicate block; The first priority is higher than the second priority.

15. The device according to claim 14, characterized in that The repeated blocks include a first repeated block and a second repeated block. When the number of blocks routed to the same node as the first repeated block is greater than the number of blocks routed to the same node as the second repeated block, the priority of the first repeated block is higher than that of the second repeated block.

16. The device according to any one of claims 10 to 15, characterized in that The deduplication block selection module is also used for: Get the target deduplication rate; Performing pre-deduplication on the duplicate blocks to determine a deduplication rate when the duplicate blocks are deduplicated according to the candidate deduplication ratio; When the deduplication rate of the duplicate blocks when being deduplicated according to the candidate deduplication rate is less than the target deduplication rate, adjusting the candidate deduplication rate until a target deduplication rate is determined that enables the deduplication rate to reach the target deduplication rate; The deduplication block selection module is specifically used for: A block to be deduplicated is determined from the multiple duplicate blocks according to the deduplication priority of each duplicate block and the target deduplication ratio.

17. The device according to any one of claims 10 to 16, characterized in that The device also includes: The writing module is used to receive a writing request, query a cache according to the fingerprint of the data block in the writing request, the fingerprint of the deduplicated block is stored in the cache, and when the fingerprint of the data block does not hit in the cache, write the data block to a persistent medium.

18. The device according to claim 17, characterized in that The writing module is also used for: When the write request includes a target data block whose fingerprint hits the cache, the frequency of the target data block is determined according to the reuse frequency of the fingerprint or the reuse distance, and the frequency includes local frequency or global frequency. According to the frequency of the target data block, the target cache algorithm adopted for the target data block is determined, and the target data block is cached according to the target cache algorithm.

19. A storage system, characterized in that: The storage system includes a controller and a memory, wherein the memory stores computer-readable instructions; the controller executes the computer-readable instructions so that the storage system executes the method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the method according to any one of claims 1 to 9.

Images