Data deduplication with perceived compression rate

Abstract

An apparatus includes a processing means configured to maintain a deduplicated data structure including sub-sections of data block identifiers associated with different compression ratio ranges and having different numbers of data block identifiers. The processing means is further configured to identify a given data block identifier and a given compression ratio for a given data block to be stored, and to determine whether the given data block identifier is in a given sub-section having a given compression ratio range including the given compression ratio. The processing means is further configured to write the given data block to a physical space block of a storage system in response to determining that the given data block identifier is not in the given sub-section, and to increment a deduplication reference count of the given data block identifier in response to determining that the given data block identifier is in the given sub-section.

Description

Technical Field

[0001] This field relates generally to information processing, and more specifically to storage in information processing systems. Background Technology

[0002] Storage arrays and other types of storage systems are typically shared by multiple host devices over a network. Applications running on these host devices each consist of one or more processes that perform application functions. These processes issue input-output (I / O) operation requests to be delivered to the storage system. The storage system's storage controller serves these I / O requests. In some information processing systems, multiple storage systems may be used to form a storage cluster. Summary of the Invention

[0003] The illustrative embodiments of this disclosure provide techniques for data deduplication that are sensitive to compression ratios.

[0004] In one embodiment, the device includes at least one processing means, the at least one processing means including a processor coupled to a memory. The at least one processing means is configured to perform the steps of: maintaining a deduplication data structure for the storage system, including data block identifiers, the deduplication data structure including two or more sub-parts associated with different compression ratio ranges, the two or more sub-parts of the deduplication data structure having different numbers of data block identifiers. The at least one processing means is further configured to perform the steps of: identifying a given data block identifier and a given compression ratio for a given data block to be stored in the storage system, and determining whether the given data block identifier of the given data block is in a given sub-part of the two or more sub-parts of the deduplication data structure having a given compression ratio range including the given compression ratio. The at least one processing means is further configured to perform the step of: in response to determining that the given data block identifier of the given data block is not in a given sub-part of the deduplication data structure, writing the given data block into a given physical space block of a plurality of physical space blocks in the storage system, the given physical space block being selected at least in part based on the given compression ratio of the given data block and the amount of unused space in the given physical space block. At least one processing device is further configured to perform the following steps: in response to determining that a given data block identifier of a given data block is in a given sub-part of a deduplication data structure, incrementing the deduplication reference count of the given data block identifier.

[0005] These and other illustrative embodiments include, but are not limited to, methods, devices, networks, systems, and processor-readable storage media. Attached Figure Description

[0006] Figure 1 This is a block diagram of an information processing system for data deduplication that is used to sense compression ratio in an illustrative implementation scheme.

[0007] Figure 2 This is a flowchart of an exemplary process for data deduplication using a sensed compression ratio in an illustrative implementation.

[0008] Figures 3A to 3C An illustrative implementation is shown that different hash entries in a deduplication hash table are used to compress data blocks with different compression ratios.

[0009] Figure 4 An example of wasted space is shown in an illustrative implementation when virtual logical blocks are mapped to physical space blocks.

[0010] Figure 5 Examples of different compression ratio levels for splitting a deduplicated hash table into multiple sub-tables are shown in the illustrative implementation.

[0011] Figure 6 The illustrated implementation shows the process flow for writing deduplicated data to a storage system using optimized deduplication hash table retention and eviction algorithms and optimized physical space block selection.

[0012] Figure 7 The illustrated implementation shows a process flow for adding cache entries to a deduplication hash table.

[0013] Figure 8 The illustrated implementation shows a process flow for removing cache entries from a deduplication hash table.

[0014] Figure 9 An example of a physical space block with used and unused space portions is shown in an illustrative implementation.

[0015] Figure 10 An example of copying a data block from one physical space block to another when the reference count of the data block exceeds the storage system limit is shown in the illustrative implementation.

[0016] Figure 11A and Figure 11B An example of a background data copying operation for reducing wasted space in physical blocks of a storage system is shown in an illustrative implementation.

[0017] Figure 12 A table showing the results of optimizations to the deduplication hash table retention and eviction algorithm for the storage system, as well as optimizations to the physical space block selection, are presented in an illustrative implementation.

[0018] Figure 13 and Figure 14 An example of a processing platform that can be used to implement at least a portion of an information processing system is shown in an illustrative implementation. Detailed Implementation

[0019] The illustrative embodiments will be described herein with reference to exemplary information processing systems and associated computers, servers, storage devices, and other processing devices. However, it should be understood that the embodiments are not limited to use with the specific illustrative system and device configurations shown. Therefore, the term "information processing system" as used herein is intended to be interpreted broadly to encompass, for example, processing systems including cloud computing and storage systems, as well as other types of processing systems including various combinations of physical and virtual processing resources. Thus, an information processing system may include, for example, at least one data center or other type of cloud-based system that includes one or more clouds hosting tenants accessing cloud resources.

[0020] Figure 1 An information processing system 100 configured according to an illustrative embodiment to provide data deduplication functionality with perceived compression ratio is illustrated. The information processing system 100 includes one or more host devices 102-1, 102-2, ..., 102-N (collectively referred to as host devices 102), which communicate via a network 104 with one or more storage arrays 106-1, 106-2, ..., 106-M (collectively referred to as storage arrays 106). The network 104 may include a storage area network (SAN).

[0021] like Figure 1 As shown, storage array 106-1 includes a plurality of storage devices 108, each storing data utilized by one or more applications running on host device 102. The storage devices 108 are illustratively arranged in one or more storage pools. Storage array 106-1 also includes one or more storage controllers 110 that facilitate input / output (I / O) processing of the storage devices 108. Storage array 106-1 and its associated storage devices 108 are examples of what is more generally referred to herein as a “storage system.” Such a storage system in embodiments of the invention is shared by host device 102 and is therefore also referred to herein as a “shared storage system.” In embodiments where only a single host device 102 exists, host device 102 may be configured to exclusively use the storage system.

[0022] Host device 102 illustratively includes a corresponding computer, server, or other type of processing device capable of communicating with storage array 106 via network 104. For example, at least a subset of host device 102 may be implemented as a corresponding virtual machine of a computing service platform or other type of processing platform. Host device 102 in such an arrangement illustratively provides computing services, such as executing one or more applications on behalf of each of one or more users associated with a corresponding host device in host device 102.

[0023] The term “user” in this article is intended to be interpreted broadly to encompass a wide range of arrangements of human, hardware, software or firmware entities, and combinations thereof.

[0024] This provides computing and / or storage services to users under Platform as a Service (PaaS), Infrastructure as a Service (IaaS), Function as a Service (FaaS), and / or Storage as a Service (STaaS) models, but it should be understood that many other cloud infrastructure deployments can be used. Furthermore, illustrative implementations can be implemented outside the context of cloud infrastructure, such as in the case of stand-alone computing and storage systems implemented within a given enterprise.

[0025] Storage device 108 of storage array 106-1 may implement logical units (LUNs) configured to store objects for a user associated with host device 102. These objects may include files, blocks, or other types of objects. Host device 102 interacts with storage array 106-1 using read and write commands, as well as other types of commands transmitted over network 104. In some embodiments, such commands more specifically include Small Computer System Interface (SCSI) commands, but other types of commands may be used in other embodiments. The terminology used extensively herein for a given I / O operation descriptively includes one or more such commands. References to terms such as “input-output” and “IO” herein should be understood as referring to input and / or output. Therefore, an I / O operation involves at least one of input and output.

[0026] Furthermore, as used herein, the term "storage device" is intended to be interpreted broadly to encompass, for example, logical storage devices, such as LUNs or other logical storage volumes. A logical storage device can be defined as a distinct portion of the storage array 106-1 that includes one or more physical storage devices. Storage device 108 can therefore be considered to include a corresponding LUN or other logical storage volume.

[0027] Storage device 108 of storage array 106-1 may be implemented using a solid-state drive (SSD). Such SSDs are implemented using non-volatile memory (NVM) devices such as flash memory. Other types of NVM devices that can be used to implement at least a portion of storage device 108 include non-volatile random access memory (NVRAM), phase-change RAM (PC-RAM), magnetic RAM (MRAM), resistive RAM (RRAM), etc. Various combinations of these and other types of NVM devices or other storage devices may also be used. For example, a hard disk drive (HDD) may be used in combination with an SSD or other types of NVM devices, or in place of an SSD or other types of NVM devices. Thus, many other types of electronic or magnetic media can be used to implement at least a subset of storage device 108. In some embodiments, it is assumed that storage array 106-1 includes persistent memory implemented using flash memory or other types of non-volatile memory of storage array 106-1. It is further assumed that the persistent memory is separate from the storage device 108 of the storage array 106-1, but in other embodiments, the persistent memory may be implemented as one or more designated portions of one or more of the storage devices 108.

[0028] In some implementations, storage array 106 may be part of a storage cluster (e.g., storage array 106 may be used to implement one or more storage nodes in a clustered storage system comprising multiple storage nodes interconnected by one or more networks), and it is assumed that host device 102 submits I / O operations for processing by the storage cluster. Different storage arrays within storage array 106 may be associated with different sites. For example, storage array 106-1 may be located at a first site, while storage array 106-2 may be located at a second site that is geographically distant from the first site.

[0029] Suppose that at least one storage controller in storage array 106 (e.g., storage controller 110 of storage array 106-1) uses compression-aware deduplication and background data movement techniques to implement functions for improving storage space utilization efficiency (e.g., across storage devices 108 of storage array 106-1, across multiple storage arrays in storage array 106 as part of a storage cluster, between a storage cluster including two or more storage arrays in storage array 106 and one or more external storage systems such as cloud-based storage 116, etc.). Such functions are provided via compression-aware deduplication module 112 and compression-aware background data movement module 114.

[0030] Compression-aware deduplication module 112 is configured to maintain a deduplication data structure, including block identifiers, for storage array 106-1 (or a storage cluster including storage array 106-1 and one or more other storage arrays in storage array 106). The deduplication data structure (e.g., a deduplication hash table) includes two or more sub-sections (e.g., sub-tables) associated with different compression ratio ranges. The two or more sub-sections of the deduplication data structure have different numbers of block identifiers. Compression-aware deduplication module 112 is also configured to identify a given block identifier and a given compression ratio for a given data block to be stored. Compression-aware deduplication module 112 is further configured to determine whether a given block identifier of a given data block is in a given sub-section of the two or more sub-sections of the deduplication data structure that has a given compression ratio range including the given compression ratio. In response to determining that a given block identifier of a given data block is not in a given sub-section of the deduplication data structure, compression-aware deduplication module 112 is configured to write the given data block to a given physical space block among multiple physical space blocks of the storage system. A given physical space block is selected at least in part based on a given compression ratio of a given data block and the amount of unused space in a given physical space block. In response to determining the given data block identifier of a given data block in a given sub-part of the deduplication data structure, the compression ratio-aware deduplication module 112 is configured to increment the deduplication reference count of the given data block identifier.

[0031] A compression-aware background data movement module 114 is configured to perform a background data replication operation to reduce the amount of unused space in a given physical space block in response to determining that the amount of unused space in a given physical space block exceeds a specified threshold. The storage system may include multiple virtual logical blocks that map logical data blocks to the physical space of the multiple physical space blocks. A given physical space block may be limited to mapping a specified number of virtual logical blocks. The background data replication operation includes migrating a first virtual logical block currently mapped to the given physical space block to another physical space block, and allocating a second virtual logical block from the multiple virtual logical blocks to the given physical space block. The first virtual logical block maps to a first logical data block having a first compression ratio, and the second virtual logical block maps to a second logical data block having a second compression ratio. The second compression ratio is lower than the first compression ratio.

[0032] Despite Figure 1In one implementation, the compression ratio-aware deduplication module 112 and the compression ratio-aware background data movement module 114 are shown to be implemented inside storage array 106-1 and outside storage controller 110. However, in other implementations, one or both of the compression ratio-aware deduplication module 112 and the compression ratio-aware background data movement module 114 may be implemented at least partially inside storage controller 110 or at least partially outside storage array 106-1, such as on one host device of host device 102, on one or more other storage arrays of storage arrays 106-2 to 106-M, or on one or more servers outside host device 102 and storage array 106 (e.g., including implementation on cloud computing platforms or other types of information technology (IT) infrastructure). Furthermore, although... Figure 1 Not shown, but other storage arrays in storage arrays 106-2 to 106-M may implement corresponding instances of the compression ratio-aware deduplication module 112 and the compression ratio-aware background data movement module 114.

[0033] At least part of the functionality of the compression ratio-aware deduplication module 112 and the compression ratio-aware background data movement module 114 can be implemented in the form of software stored in memory and executed by a processor.

[0034] As described above, it is assumed that storage array 106 in some embodiments is part of a storage cluster. A storage cluster can provide or implement multiple different storage tiers in a multi-tiered storage system. As an example, a given multi-tiered storage system may include a speed or performance tier implemented using flash memory devices or other types of SSDs, and a capacity tier implemented using HDDs; one or more of these tiers may be server-based. It will be apparent to those skilled in the art that a variety of other types of storage devices can be used in other embodiments, and the specific storage devices used in a given storage tier of a multi-tiered storage system may vary depending on the specific needs of a given embodiment, and a variety of different storage device types may be used within a single storage tier. As previously indicated, the term "storage device" as used herein is intended to be interpreted broadly and therefore may encompass, for example, SSDs, HDDs, flash drives, hybrid drives, or other types of storage products and devices or portions thereof, and illustratively includes logical storage devices such as LUNs.

[0035] It should be understood that a tiered storage system may include more than two storage tiers, such as one or more "performance" tiers and one or more "capacity" tiers, wherein the performance tiers illustratively provide increased IO performance characteristics relative to the capacity tiers, and the capacity tiers illustratively use storage with relatively lower cost than the performance tiers. There may also be multiple performance tiers, each providing a different level of service or performance as needed, or multiple capacity tiers.

[0036] Figure 1 The host device 102 and storage array 106 in the embodiments are assumed to be implemented using at least one processing platform, wherein each processing platform includes one or more processing units, each processing unit having a processor coupled to memory. Such processing units may illustratively include specific arrangements of computing, storage, and networking resources. For example, in some embodiments, the processing units are implemented at least in part using virtual resources such as virtual machines (VMs) or Linux containers (LXCs) or a combination of both, such as in an arrangement in which Docker containers or other types of LXCs are configured to run on VMs.

[0037] Although the host device 102 and the storage array 106 can be implemented on correspondingly different processing platforms, many other arrangements are possible. For example, in some embodiments, at least a portion of one or more host devices 102 and one or more storage arrays 106 are implemented on the same processing platform. One or more storage arrays of the storage array 106 can therefore be implemented, at least partially, within at least one processing platform that implements at least a subset of the host device 102.

[0038] Network 104 may be implemented using multiple different types of networks to interconnect storage system components. For example, network 104 may include a SAN as part of a global computer network such as the Internet, but other types of networks may be part of a SAN, including wide area networks (WANs), local area networks (LANs), satellite networks, telephone or wired networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or various portions or combinations of these and other types of networks. Therefore, in some embodiments, network 104 includes a combination of multiple different types of networks, each of which includes processing means configured to communicate using Internet Protocol (IP) or other related communication protocols.

[0039] As a more specific example, some implementations may utilize one or more high-speed local area networks (LANs), in which associated processing devices communicate with each other using peripheral component Fast Interconnect (PCIe) cards and networking protocols such as Unlimited Bandwidth, Gigabit Ethernet, or Fibre Channel. As those skilled in the art will appreciate, many alternative networking arrangements are possible in a given implementation.

[0040] While in some embodiments certain commands used by host device 102 to communicate with storage array 106 illustratively include SCSI commands, other types of commands and command formats may be used in other embodiments. For example, some embodiments may utilize command features and functionalities associated with NVM Fast (NVMe) to perform I / O operations, as described in the NVMe specification, revision 2.0a, July 2021, which is incorporated herein by reference. Other storage protocols of this type that may be utilized in the illustrative embodiments disclosed herein include NVMe over Fabric (also known as NVMeoF) and NVMe over Transmission Control Protocol (TCP) (also known as NVMe / TCP).

[0041] As described above, communication between the host device 102 and the storage array 106 can utilize a PCIe connection or other types of connections implemented through one or more networks. For example, illustrative embodiments may use interfaces such as Internet SCSI (iSCSI), Serial Attached SCSI (SAS), and Serial ATA (SATA). In other embodiments, many other interfaces and associated communication protocols may be used.

[0042] In some implementations, storage array 106 may be implemented as part of a cloud-based system.

[0043] Therefore, it should be understood that, as used herein, the term “memory array” is intended to be interpreted broadly and may encompass several different instances of commercially available memory arrays.

[0044] Other types of storage products that can be used to implement the given storage system in the illustrative embodiments include software-defined storage, cloud storage, object-based storage, and scale-out storage. In the illustrative embodiments, combinations of these and several other storage types can also be used to implement the given storage system.

[0045] In some implementations, the storage system includes a first storage array and a second storage array arranged in an active-active configuration. For example, such an arrangement can be used to ensure that data stored in one of the storage arrays is replicated to the other storage array using a synchronous replication process. Such data replication across multiple storage arrays can facilitate fault recovery in system 100. Thus, one storage array can operate as a production storage array relative to another storage array operating as a backup or recovery storage array.

[0046] However, it should be understood that the embodiments disclosed herein are not limited to active-active configurations or any other particular storage system arrangement. Therefore, the illustrative embodiments described herein can be configured using a variety of other arrangements, including, for example, active-passive arrangements, active-active asymmetric logical unit access (ALUA) arrangements, and other types of ALUA arrangements.

[0047] These and other storage systems may be part of what is more generally referred to herein as a processing platform, which includes one or more processing units, each including a processor coupled to memory. A given such processing unit may correspond to one or more virtual machines or other types of virtualization infrastructure, such as Docker containers or other types of LXC. As described above, communication between such components of system 100 may be via one or more networks.

[0048] As used herein, the term "processing platform" is intended to be interpreted broadly to encompass (by way of illustration and not limitation) multiple sets of processing devices and one or more associated storage systems configured to communicate over one or more networks. For example, a distributed implementation of host device 102 is possible, wherein some host devices of host device 102 reside in a data center located in a first geographic location, while other host devices of host device 102 reside in one or more other data centers located in one or more other geographic locations that may be far from the first geographic location. Storage array 106 may be implemented at least partially in the first geographic location, the second geographic location, and one or more other geographic locations. Thus, in some implementations of system 100, different host devices of host device 102 and storage array 106 may reside in different data centers.

[0049] Numerous other distributed implementations of host device 102 and storage array 106 are possible. Therefore, host device 102 and storage array 106 can also be implemented in a distributed manner across multiple data centers.

[0050] The following will combine Figure 13 and Figure 14 Additional examples of the processing platform used to implement part of system 100 in the illustrative implementation are described in more detail.

[0051] It should be understood that Figure 1 The specific set of elements shown for data deduplication to perceive compression ratio are presented as illustrative examples only, and in other embodiments, additional or alternative elements may be used. Thus, another embodiment may include different arrangements of additional or alternative systems, devices and other network entities, as well as modules and other components.

[0052] It should be understood that these and other features of the illustrative implementation are presented by way of example only and should not be construed as limiting in any way.

[0053] Now refer to Figure 2 The flowchart describes in more detail an exemplary process for data deduplication using a sensed compression ratio. It should be understood that this particular process is merely an example, and additional or alternative processes for data deduplication using a sensed compression ratio may be used in other embodiments.

[0054] In this implementation, the process includes steps 200 to 208. These steps are assumed to be performed by a compression-aware deduplication module 112 and a compression-aware background data movement module 114. The process begins at step 200: maintaining a deduplication data structure including data block identifiers for the storage system. The deduplication data structure includes two or more sub-parts associated with different compression ratio ranges. The two or more sub-parts of the deduplication data structure have different numbers of data block identifiers. In step 202, a given data block identifier and a given compression ratio are identified for a given data block to be stored in the storage system.

[0055] Figure 2 The process continues to step 204: determining whether the given data block identifier of a given data block is in a given sub-section of two or more sub-sections of the deduplication data structure, having a given compression ratio range including a given compression ratio. In response to determining that the given data block identifier of a given data block is not in a given sub-section of the deduplication data structure, in step 206, the given data block is written to a given physical space block among multiple physical space blocks of the storage system. The given physical space block is selected at least in part based on the given compression ratio of the given data block and the amount of unused space in the given physical space block. In response to determining that the given data block identifier of a given data block is in a given sub-section of the deduplication data structure, in step 208, the deduplication reference count of the given data block identifier is incremented.

[0056] The deduplication data structure may include a deduplication hash table, and the data block identifier includes a hash of the data block content. Each of the two or more sub-parts of the deduplication data structure may have a target length calculated at least in part based on its associated compression ratio range.

[0057] Step 200 may include, in response to receiving a new data block identifier associated with a new data block to be added to the deduplication data structure, determining whether the deduplication data structure has available space for the new data block identifier. Determining whether the deduplication data structure has available space for the new data block identifier may include determining whether the sum of data block identifiers across two or more sub-sections of the deduplication data structure is less than the target total number of data block identifiers in the deduplication data structure. In response to determining that the deduplication data structure has available space for the new data block identifier, identifying the compression ratio of the new data block, and adding the new data block identifier to one sub-section of the two or more sub-sections of the deduplication data structure that has an associated compression ratio range including the identifier of the new data block.

[0058] In response to determining that the deduplicated data structure does not have available space for a new data block identifier, one or more existing data block identifiers are evicted from the deduplicated data structure, and a new data block identifier is added to one of two or more sub-parts of the deduplicated data structure, which has an associated compression ratio range with a compression ratio including the identifier of the new data block. Each of the two or more sub-parts of the deduplicated data structure may have a target length calculated at least in part based on its associated compression ratio range, and evicting one or more existing data block identifiers from the deduplicated data structure may include evicting one or more existing data block identifiers from sub-parts of the deduplicated data structure where the number of entries exceeds its associated target length. Evicting one or more existing data block identifiers from the deduplicated data structure may also include evicting one or more existing data block identifiers from sub-parts of the deduplicated data structure where the number of entries exceeds its associated target length, and these entries are either least recently used or least frequently used.

[0059] Figure 2 The process also includes, in response to determining that the amount of unused space in a given physical space block exceeds a specified threshold, performing a background data replication operation to reduce the amount of unused space in the given physical space block. The storage system may include multiple virtual logical blocks that map logical data blocks to the physical space of the multiple physical space blocks. A given physical space block may be limited to mapping a specified number of multiple virtual logical blocks. The background data replication operation may include migrating a first virtual logical block currently mapped to the given physical space block to another physical space block, and allocating a second virtual logical block from the multiple virtual logical blocks to the given physical space block. The first virtual logical block maps to a first logical data block having a first compression ratio, and the second virtual logical block maps to a second logical data block having a second compression ratio. The second compression ratio is lower than the first compression ratio.

[0060] Compression and deduplication are two techniques that can provide significant space savings in storage systems. Compression aims to limit the amount of storage capacity used by reducing the actual size of the data stored. Deduplication reduces the amount of storage capacity used by limiting the same dataset that consumes storage space to a single (or fewer) instance. Deduplication methods can divide data into small blocks and assign a unique identifier (e.g., a unique hash identifier) ​​to each block. Deduplication methods can utilize inline deduplication and / or post-deduplication. Inline deduplication is performed during data transfer to storage, where the deduplication algorithm checks the hash identifier to see if it already exists in storage. If the hash identifier exists, the new copy is not stored in physical storage. However, inline deduplication is difficult to perform on all incoming data blocks without impacting performance. Post-deduplication is performed after data has been written to storage without deduplication, where data is read from physical storage to check if its hash identifier already exists in storage. If it exists, the data is not stored on physical storage.

[0061] Deduplication methods utilize deduplication hash tables (e.g., in deduplication caches) to store identifiers (e.g., hash identifiers) for data blocks. Only data blocks whose identifiers "hit" entries in the deduplication hash table have a chance to be deduplicated (removing duplicate data). However, the number of hash entries is finite because the size of the deduplication hash table is limited for performance and memory considerations. To improve deduplication efficiency, the technical solutions described in this paper optimize or improve cache replacement or eviction algorithms to retain only entries with a high deduplication probability level in the deduplication hash table.

[0062] In some implementations, the impact of the compression ratio characteristics of different data blocks on storage system efficiency is considered to improve storage system efficiency (e.g., fully utilize storage space or improve the use of storage space). There are various technical problems that prevent storage systems from fully utilizing available storage space, including: (1) the compression ratio of data blocks with entries in the deduplication hash table affects data deduplication efficiency; and (2) the storage system can report insufficient space, but when the workload has data that can be compressed beyond a certain specified threshold (e.g., more than 8:1), storage space utilization may be less than 100%, making mitigation actions necessary to reduce wasted unusable space.

[0063] Regarding technical issue (1), since the size of the deduplication hash table may be finite, it is difficult to retain deduplication hash table entries for all data blocks. In some cases, deduplication is used in conjunction with data compression to save storage space. The compression ratio value of (deduplicatable) data blocks that can remove duplicate data can be varied. To illustrate how the data compression ratio affects deduplication by retaining entries in the deduplication hash table, five types of data blocks with different compression ratios (e.g., 5:1, 4:1, 3:1, 2:1, 1:1) are considered as examples. Figure 3A Data blocks 302-1, 302-2, 302-3, 302-4 and 302-5 (collectively referred to as data block 302) with such different compression ratios are shown, and the associated size of data block 302 before compression is shown. Figure 3A The data size of compressed data block 302 is also shown, which is visualized as compressed data blocks 304-1, 304-2, 304-3, 304-4 and 304-5 (collectively referred to as compressed data block 304).

[0064] To eliminate the influence of other factors, it is assumed that data blocks 302 have equal deduplication probability levels (e.g., data blocks 302 have equal chances of being deduplicated). Figure 3B and Figure 3C A set of data 306 to be compressed is shown, comprising multiple instances of each of data blocks 302. Figure 3B and Figure 3C In both examples, it is assumed that the deduplication hash table is limited to three entries. For Figure 3B The deduplication hash entry 308-1 is used for data blocks 302-1, 302-2, and 302-3. For Figure 3C Deduplication hash entry 308-2 is used for data blocks 302-3, 302-4, and 302-5. Therefore, deduplication hash entries 308-1 and 308-2 are used for data blocks in data block 302 with different compression ratios, meaning the only factor affecting the compressed data size is the compression ratio of data block 302. For example... Figure 3B and Figure 3C As shown, the compressed data 310-1 using deduplication hash entry 308-1 is larger than the compressed data 310-2 using deduplication hash entry 308-2. The compressed data 310-1 and 310-2 illustrate the data on the disk after compression, or how much physical space is needed to store the dataset 306 to be compressed, which includes both deduplicated and undeduplicated blocks (e.g., according to deduplication hash entries 308-1 and 308-2). Although in Figure 3B and Figure 3C In the example, deduplication hash entries 308-1 and 308-2 have the same deduplication hit rate, but the physical space allocated to compressed data 310-2 is smaller than that allocated to compressed data 310-1. This means that the deduplication space of compressed data 310-2 is greater than that of compressed data 310-1. Figure 3B and Figure 3C This demonstrates that when data blocks have equal deduplication probability levels, retaining data blocks with lower compression rates in the deduplication hash table can help improve deduplication efficiency.

[0065] Regarding technical issue (2), when implementing data reduction features, the storage system can utilize virtual entries to map logical space to physical space. Multiple virtual entries can reside in a single Virtual Logical Block (VLB), and multiple VLBs can refer to a unit of physical space (e.g., a physical space block). However, the number of VLBs that can be mapped to a single physical space block may be limited by the storage system, as a larger number will result in wasted VLBs (e.g., metadata space loss) and make the implementation more complex, while a smaller number will result in wasted physical space (e.g., user data space). In the various implementations described below, the limit is assumed to be 8:1 (e.g., 8 VLBs can refer to a single physical space block). In some cases, when the workload has data that can be compressed beyond 8:1, the storage system may report an "out of space" condition with space utilization less than 100%. Figure 4 An example of this implementation is shown, with a set of VLBs 401-1, 401-2, ..., 401-V (collectively referred to as VLB 401), which map to the same physical space block 403. Here, it is assumed that V is 8. Each of the VLBs 401 has a maximum data size X in megabytes (MB), and the physical space block 403 also has a size of X MB per unit. Figure 4 In the example, physical space block 403 is already referenced by the maximum number of VLBs 401 (e.g., 8 VLBs), and the compressed data has a compression ratio greater than 8:1, which results in physical space block 403 having unusable wasted space 430, into which no new user data can be written.

[0066] The illustrative implementation overcomes the aforementioned technical problems by adaptively using variable-length sub-tables within the deduplication hash table, where the sub-tables are data block compression rate aware. Therefore, entries with identifiers of data blocks with lower compression rates have a greater chance of being retained in the deduplication hash table. This further reduces the capacity requirements of the storage system for storing the same amount of data. The optimization described herein helps to fully utilize or improve the use of physical space by allocating copies of lower-compression-rate data blocks to previously unavailable, wasted space via background data replication operations. A novel approach is provided based on the partitioning of compression rate levels and the constraint of grouping entries with optimal lengths based on compression rate levels, which is effective in optimizing deduplication hash table retention and eviction policies. This provides a corresponding improvement in storage space utilization efficiency. Advantageously, the technique described herein is lightweight and utilizes appropriately modified cache retention and eviction algorithms for deduplication cache table entries to separate cache entries with higher deduplication rates from those with lower deduplication rates. In other words, the cache retention and eviction algorithm is modified to give lower-compression-rate data blocks a greater chance of being retained in the deduplication hash table, thereby improving storage space utilization efficiency.

[0067] In some implementations, the deduplication hash table is divided into multiple sub-tables of variable length based on the actual physical space consumed by the data blocks (e.g., which can typically be determined by the data compression rate of the data blocks). Figure 5 Example 500 of this type of partitioning is shown, where the data block compression ratio is divided into five levels: 501-1, 501-2, 501-3, 501-4, and 501-5. By partitioning the data blocks into compression ratio levels, two optimization points are provided to address [the issue / problem]. Figure 6 The above-mentioned technical problems (1) and (2) in process flow 600. In process flow 600, the incoming data block to be written is received in step 601, and then data deduplication is performed in step 602A. The data deduplication step 602A provides a first optimization point via step 602B, wherein the deduplication hash table cache entry retention and eviction algorithm is optimized in consideration of the above-mentioned technical problem (2). In step 603, data blocks that do not "hit" cache entries in the deduplication hash table are identified, and such data blocks are written to physical space in step 604A. The physical space writing step 604A provides a second optimization point via step 604B, wherein the physical space block selection is optimized in consideration of the above-mentioned technical problem (1). In step 605, the cache entries in the deduplication hash table are updated.

[0068] The following symbols are used in the description below:

[0069] C is used to represent the compression ratio of the data block, where C = size_original / size_after_compression;

[0070] Length_total is used to represent the maximum number of hash table entries used for deduplication;

[0071] Subtable_c[1] is used to represent the deduplication hash subtable, which includes identifiers of data blocks with a compression ratio of level 1 C. Figure 5 In the example, level 501-1), where C ≤ 8:6;

[0072] Subtable_c[2] is used to represent the deduplication hash subtable, which includes identifiers of data blocks with a compression ratio of level 2 C. Figure 5 (Example: Level 501-2), where 8:6 <C≤8:4;

[0073] Subtable_c[3] is used to represent the deduplication hash subtable, which includes identifiers of data blocks with a compression ratio of 3 C. Figure 5 In the example, level 501-3), where 8:4 <C≤8:2;

[0074] Subtable_c[4] is used to represent the deduplication hash subtable, which includes identifiers of data blocks with a compression ratio of 4 C. Figure 5 In the example, level 501-4, the ratio is 8:2. <C≤8:1;

[0075] Subtable_c[5] is used to represent the deduplication hash subtable, which includes identifiers of data blocks with a compression ratio of 5 C. Figure 5 In the example, level 501-5), where C > 8:1;

[0076] L[1] 最佳 Used to represent the target length of subtable _c[1], which is the optimal number of entries in subtable _c[1], where

[0077] L[2] 最佳 Used to represent the target length of subtable _c[2], which is the optimal number of entries in subtable _c[2], where

[0078] L[3] 最佳 Used to represent the target length of subtable _c[3], which is the optimal number of entries in subtable _c[3], where

[0079] L[4] 最佳 Used to represent the target length of subtable _c[4], which is the optimal number of entries in subtable _c[4], where

[0080] L[5] 最佳 Used to represent the target length of subtable _c[5], which is the optimal number of entries in subtable _c[5], where

[0081] L is used to represent a list recording the actual length of each sub-table, where L[n] is the number of entries in sub-table _c[n]; and

[0082] M is used to represent the variable number of entries used in the deduplication hash table entry eviction algorithm.

[0083] Considering the aforementioned technical problem (1), an optimization of the deduplication hash table entry retention and eviction algorithm will now be described (e.g., Figure 6 (Optimization point of step 604B in process flow 600). In some implementations, the retention and eviction algorithm for deduplicating hash table cache entries is used to separate cache entries with higher deduplication rates from those with lower deduplication rates. Various types of retention and eviction algorithms can be used, including Least Frequently Used (LFU), Least Recently Used (LRU), etc. There may be a large deduplicating hash table that holds all entries, and when a new incoming entry is added to the deduplicating hash table but the deduplicating hash table is full, the least frequently used or least recently used entry is evicted. In some implementations, such algorithms are optimized by dividing a large deduplicating hash table into multiple sub-tables of variable length, with the constraint that the total number of entries (e.g., the sum of all entries in the sub-tables) should not exceed the total length. Figure 7 The process flow 700 of the hash entry addition algorithm is shown; and Figure 8 The flowchart 800 of the hash entry eviction algorithm is shown. For example... Figure 7 and Figure 8 As shown, it is not necessary to record the compression ratio of data blocks in the deduplication hash table—it may only be necessary to query the compression ratio of data blocks when creating an entry and adding it to its corresponding sub-table.

[0084] Figure 7 The process flow 700 begins in step 701, and in step 703, a new entry is received, which has an identifier for a data block x with a compression ratio C[x]. In step 705, it is determined whether the sum (L) is less than the length_total. If the result of step 705 is no, then the cache entry eviction algorithm (e.g., Figure 8 The process flow 800 is triggered to evict old entries from the deduplication hash table in step 707. If the result determined in step 705 is yes, then in step 709, the new cache entry is added to the sub-table _c[n] to which the compression ratio C[x] belongs, and then process flow 700 ends in step 711.

[0085] Figure 8The process flow 800 begins in step 801, and in step 803, the M entries with the minimum deduplication probability are identified in the deduplication hash table: e[m], m = {1, 2, ..., M}. In step 805, for each e[m], m = {1, 2, ..., M}, if e[m] belongs to sublist _c[n] and L[n] ≥ L[n], ... 最佳 If e[m] is evicted from sub-table _c[n], and L[n] is updated to L[n]-1. In step 807, it is determined whether the sum (L) < length_total. If the result of step 807 is no, the process proceeds to step 809, where the least recently used (or least frequent) entry e[1] is evicted, and then process flow 800 ends in step 811. If the result of step 807 is yes, then process flow 800 ends in step 811.

[0086] Figure 7 and Figure 8 The optimized hash entry retention and eviction algorithm utilizes variable-length sub-tables that are partitioned by leveraging the compression ratio awareness of data blocks. Therefore, entries with lower compression ratio block identifiers have a greater chance of being retained in the deduplication hash table. This helps save physical space and improve data deduplication efficiency, as shown below compared to... Figure 12 Further detailed description.

[0087] Considering the aforementioned technical problem (2), an optimization of the physical space block selection algorithm will now be described (e.g., Figure 6 (Optimization point of step 602B in process flow 600). In the following description, "data block" is used to refer to a data block represented by deduplicated hash table entries. VLB is used to map logical data blocks to physical space, where the maximum total logical data size is XMB. Physical space block is used to represent a physical space unit provided by the storage system for storing data of size XMB. PB represents a set of physical space blocks used to record those that already have up to 8 (e.g., the limit in this example) VLBs mapped to them, but have a wasted space ratio greater than 1 / 8. ws[i] is used to represent the wasted space ratio of physical space block i in PB. For example, Figure 9 A physical space block 903 is shown, having 3 / 4 of the space used 931 and 1 / 4 of the space wasted 933 (e.g., physical space block 903 has ws[i] = 1 / 4).

[0088] As an example, consider the deduplication reference count of the same data block n that has reached a maximum limit (e.g., 256). When the reference count reaches the limit, data block n is copied as a new data block, and its hash entry is updated to record the identifier of the newly copied data block. Figure 10The example illustrates this, where a set of VLBs 1001-1-1 to 1001-1-V (collectively referred to as VLB 1001-1) are mapped to physical space block 1003-1. When the deduplication reference count of a specific data block n in VLB 1001-1-V reaches its limit, the data block is copied to a new VLB 1001-2 and the corresponding physical space block 1003-2.

[0089] In some implementations, algorithms are used to utilize wasted space in the PB to store low-compression copies of sub-tables with lower compression ratios (e.g., sub-table_c[1], sub-table_c[2], sub-table_c[3], sub-table_c[4]). Figure 11A and Figure 11B An example of such an algorithm is shown, in which a set of 1101-1-1, 1101-1-2, ..., 1101-1-V (collectively referred to as VLB1101-1) is mapped to physical space block 1103-1. For physical space block 1103-1 (PB[i]), one of its corresponding VLBs (VLB 1101-1-V) is moved to another VLB 1101-2 mapped to another physical space block 1103-2. Then, VLB1101-1-V and its relationship with physical space block 1103-1 (PB[i]) are cleared, such that physical space block 1103-1 (PB[i]) has fewer than the maximum number of referenced VLBs (e.g., the maximum number V of referenced VLBs could be 8). A new VLB, denoted as VLB1101-1-V', is allocated to physical space block 1103-1 and used to create a mapping from the newly copied low-compression data blocks to physical space block 1103-1. This results in the waste space of physical space block 1103-1 being improved (e.g., reduced) from 1130-1 to 1130-2, as... Figure 11A and Figure 11B As shown. When physical space block 1103-1 (PB[i]) cannot allocate enough space for the new replica, another physical space block 1103-2 (PB[j]) is selected to allocate space for the new replica. As mentioned above, one of the original VLBs of physical space block 1103-2 (PB[j]) can be migrated first.

[0090] This optimization helps to fully utilize or improve the use of physical space in the storage system by allocating copies of data blocks with lower compression ratios to previously wasted space. Advantageously, this approach does not impact user write I / O performance because it is a background data copying operation, and may result in more read I / O hits on physical space blocks due to the higher likelihood of deduplication.

[0091] Now relative to Figure 12An exemplary implementation of the technique described herein is described in more detail. First, data source preparation is performed. For simplicity and clarity, 8-kilobyte (KB) granular data blocks are generated from dataset F, where F = {f1, f2, f3, ..., f200}, and the compression ratio C is randomly selected from the set {[8:8], [8:6], [8:3], [8:1], [16:1]} corresponding to each type of data block. For initialization, the maximum size of the deduplication hash table is set to length_total = 100. Therefore, the target length of each sub-table is L[1]. 最佳 =40, L[2] 最佳 =29、L[3] 最佳 =18、L[4] 最佳 =8、L[5] 最佳 =2. For an incoming deduplication query, the deduplication hash table entries are checked for matching. If no matching entry is found, the data is stored. Otherwise, a reference count is added to the previously stored data block. Here, the maximum deduplication reference count for the same stored data block is assumed to be 256. Data blocks that reach the reference count limit are copied to consume wasted space in the physical data block in the PB. The deduplication hash table retention and eviction process is then triggered. At each time interval, if the deduplication hash table has no space for a new entry, a new entry to be added to the deduplication hash table is generated and used to trigger the eviction iteration (e.g., based on...). Figure 7 Process flow 700 and Figure 8 The process flow is as described above (800). A 1 terabyte (TB) I / O write is simulated, with the results shown in... Figure 12 In Table 1200. As shown in Table 1200, the optimization method described in this paper significantly improves deduplication efficiency by reducing physical space consumption by approximately 15%, wherein:

[0092]

[0093] Physical space Δ = Allocated physical space (original) - Allocated physical space (optimized)

[0094] exist Figure 12 In the example, the deduplication hash table entry eviction algorithm used is LFU, where the least frequently used entries are discarded first. A similar approach, based on LFU, is used to identify entries with the lowest deduplication probability and employs variable-length sub-tables that leverage the compression rate-aware partitioning of data blocks. However, in other implementations, LRU or other types of cache eviction algorithms may be used.

[0095] The solution described in this paper offers various advantages over conventional methods. For example, the solution takes into account the compression characteristics of data blocks and leverages the impact of compression on deduplication efficiency, which reduces the capacity requirements of the storage system for storing the same amount of data. Furthermore, the solution results in less data being written to the physical drives, which advantageously reduces wear and extends the lifespan of the physical drives (e.g., reducing the wear level of flash memory in SSDs). This provides cost savings, as solid-state storage is expensive. The solution also helps to use less physical capacity, enabling the storage system to serve more user data.

[0096] It should be understood that the specific advantages described above and elsewhere herein are associated with specific illustrative embodiments and are not required to exist in other embodiments. Moreover, the features and functionality of the specific type of information processing system, as shown in the accompanying drawings and as described above, are merely exemplary, and numerous other arrangements may be used in other embodiments.

[0097] Now refer to Figure 13 and Figure 14 An illustrative embodiment of a processing platform for achieving data deduplication with perceived compression ratio is described in more detail. Although described in the context of system 100, these platforms may also be used to implement at least a portion of other information processing systems in other embodiments.

[0098] Figure 13 An example processing platform including cloud infrastructure 1300 is shown. Cloud infrastructure 1300 includes components that can be used to implement... Figure 1 The information processing system 100 comprises at least a portion of its physical and virtual processing resources. The cloud infrastructure 1300 includes multiple virtual machines (VMs) and / or container groups 1302-1, 1302-2, ..., 1302-L implemented using virtualization infrastructure 1304. Virtualization infrastructure 1304 runs on physical infrastructure 1305 and illustratively includes one or more hypervisors and / or operating system-level virtualization infrastructures. Operating system-level virtualization infrastructure illustratively includes the kernel control group of a Linux operating system or other types of operating systems.

[0099] The cloud infrastructure 1300 also includes multiple application groups 1310-1, 1310-2, ..., 1310-L, which run on corresponding VM / container groups 1302-1, 1302-2, ..., 1302-L under the control of the virtualization infrastructure 1304. A VM / container group 1302 may include a corresponding VM, one or more containers of the corresponding group, or one or more containers of the corresponding group running within a VM.

[0100] exist Figure 13 In some implementations of the scheme, VM / container group 1302 includes corresponding VMs implemented using virtualization infrastructure 1304 including at least one hypervisor. A hypervisor platform can be used to implement the hypervisor within virtualization infrastructure 1304, wherein the hypervisor platform has an associated virtual infrastructure management system. The underlying physical machines may include one or more distributed processing platforms, which include one or more storage systems.

[0101] exist Figure 13 In other implementations of the scheme, VM / container group 1302 includes corresponding containers implemented using virtualization infrastructure 1304 that provides operating system-level virtualization functionality, such as Docker containers supporting operation on bare metal hosts or Docker containers running on VMs. The containers are implemented illustratively using the corresponding kernel control group of the operating system.

[0102] As is evident from the above, one or more of the processing modules or other components of system 100 may each run on a computer, server, storage device, or other processing platform element. Such a given element may be considered as an example of what is more generally referred to herein as a "processing device". Figure 13 The cloud infrastructure 1300 shown may represent at least a portion of a processing platform. Another example of such a processing platform is... Figure 14 The processing platform shown is 1400.

[0103] In this embodiment, the processing platform 1400 includes a portion of the system 100 and includes a plurality of processing devices, referred to as 1402-1, 1402-2, 1402-3, ... 1402-K, which communicate with each other via a network 1404.

[0104] Network 1404 may include any type of network, such as global computer networks (such as the Internet), WAN, LAN, satellite networks, telephone or cable networks, cellular networks, wireless networks (such as WiFi or WiMAX networks), or various parts or combinations of these and other types of networks.

[0105] The processing device 1402-1 in the processing platform 1400 includes a processor 1410 coupled to a memory 1412.

[0106] Processor 1410 may include a microprocessor, microcontroller, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), video processing unit (VPU), or other types of processing circuitry, as well as portions or combinations of such circuitry elements.

[0107] Memory 1412 may include random access memory (RAM), read-only memory (ROM), flash memory, or other types of memory in any combination. Memory 1412 and other memories disclosed herein should be considered as illustrative examples of what is more generally referred to as a “processor-readable storage medium” storing executable program code of one or more software programs.

[0108] Articles of manufacture including such processor-readable storage media are considered illustrative embodiments. Given such articles of manufacture may include, for example, storage arrays, storage disks, or integrated circuits containing RAM, ROM, flash memory, or other electronic memory, or any of a variety of other types of computer program products. As used herein, the term "article of manufacture" should be understood to exclude transient propagated signals. Many other types of computer program products including processor-readable storage media may be used.

[0109] The processing device 1402-1 also includes a network interface circuit 1414 for interfacing the processing device with the network 1404 and other system components, and may include a conventional transceiver.

[0110] It is assumed that the other processing devices 1402 of the processing platform 1400 are configured in a manner similar to that shown for the processing device 1402-1 in the figure.

[0111] Similarly, the particular processing platform 1400 shown in the figures is presented by way of example only, and the system 100 may include additional or alternative processing platforms, as well as a number of different processing platforms in any combination, wherein each such platform includes one or more computers, servers, storage devices or other processing devices.

[0112] For example, other processing platforms used to implement the illustrative implementation scheme may include converged infrastructure.

[0113] Therefore, it should be understood that in other embodiments, different arrangements of additional or alternative elements may be used. At least a subset of these elements may be implemented together on a common processing platform, or each such element may be implemented on a separate processing platform.

[0114] As previously described, components of the information processing system disclosed herein can be implemented, at least in part, as one or more software programs stored in memory and executed by a processor of a processing device. For example, the data deduplication function with perceived compression ratio disclosed herein is at least partially implemented, illustratively, in the form of software running on one or more processing devices.

[0115] It should be emphasized again that the above embodiments are provided for illustrative purposes only. Many variations and other alternative embodiments can be used. For example, the disclosed technology is applicable to a variety of other types of information processing systems, storage systems, etc. Moreover, the specific configurations of the system and apparatus elements illustratively shown in the drawings, and the associated processing operations, may change in other embodiments. Furthermore, the various assumptions made above in describing the illustrative embodiments should be considered exemplary and not as requirements or limitations of this disclosure. Numerous other alternative embodiments within the scope of the appended claims will be apparent to those skilled in the art.

Claims

1. An apparatus, the apparatus comprising: At least one processing device, the at least one processing device including a processor coupled to a memory; The at least one processing device is configured to perform the following steps: To maintain a storage system a deduplicated data structure including a target total number of data block identifiers, the deduplicated data structure includes two or more sub-parts associated with different compression ratio ranges, the two or more sub-parts of the deduplicated data structure having corresponding target number of entries, the sum of the target number of the two or more sub-parts of the deduplicated data structure being less than or equal to the total number of target entries of the deduplicated data structure, the two or more sub-parts of the deduplicated data structure including a first sub-part and at least a second sub-part, the first sub-part having a first target number of data block identifiers for data blocks with a compression ratio in a first compression ratio range, the second sub-part having a second target number of data block identifiers for data blocks with a compression ratio in a second compression ratio range, the second target number of entries being different from the first target number of entries, and the second compression ratio range being different from the first compression ratio range; For a given data block to be stored in the storage system, identify the given data block identifier and the given compression ratio; Determine whether the given data block identifier is in a given sub-part of the two or more sub-parts of the deduplication data structure that has a given compression ratio range including the given compression ratio; In response to determining that the given data block identifier of the given data block is not in the given sub-part of the deduplication data structure, (i) the given data block is written to a given physical space block among a plurality of physical space blocks of the storage system, the given physical space block being selected at least in part based on the given compression ratio of the given data block and the amount of unused space in the given physical space block, and (ii) whether to add the given data block identifier as an entry to the given sub-part of the deduplication data structure, based at least in part on whether the number of allocated entries in the given sub-part of the deduplication data structure is less than the given target number of entries in the given sub-part of the deduplication data structure; as well as In response to determining the given data block identifier in the given sub-part of the deduplication data structure, the deduplication reference count of the given data block identifier is incremented.

2. The device of claim 1, wherein the deduplication data structure comprises a deduplication hash table, and wherein the data block identifier comprises a hash of the data block content.

3. The device of claim 1, wherein the number of the given target entries of the given sub-part of the deduplication data structure is calculated at least in part based on the given compression rate range associated with the given sub-part of the deduplication data structure.

4. The device of claim 1, wherein maintaining the deduplication data structure includes, in response to receiving a new data block identifier associated with a new data block for adding to the deduplication data structure, determining whether the deduplication data structure has available space for the new data block identifier.

5. The apparatus of claim 4, wherein determining whether the deduplication data structure has available space for the new data block identifier comprises: Determine whether the sum of allocated entries across the two or more sub-parts of the deduplication data structure is less than the target total number of entries for the data block identifier of the deduplication data structure.

6. The apparatus of claim 4, wherein maintaining the deduplication data structure further comprises, in response to determining that the deduplication data structure has available space for the new data block identifier: The compression ratio of the new data block is identified; and Add the new data block identifier to one of the two or more sub-parts of the deduplication data structure, which has an associated compression ratio range that includes the identified compression ratio of the new data block.

7. The apparatus of claim 4, wherein maintaining the deduplication data structure further comprises, in response to determining that the deduplication data structure does not have available space for the new data block identifier: Extract one or more existing data block identifiers from the deduplication data structure; and Add the new data block identifier to one of the two or more sub-parts of the deduplication data structure, which has an associated compression ratio range that includes the identified compression ratio of the new data block.

8. The device of claim 7, wherein removing the one or more existing data block identifiers from the deduplication data structure comprises: The one or more existing data block identifiers are evicted from the sub-parts of the two or more sub-parts of the deduplication data structure whose number of allocated entries exceeds the number of their associated target entries.

9. The device of claim 8, wherein removing the one or more existing data block identifiers from the deduplication data structure comprises: The one or more existing data block identifiers are evicted from the sub-parts of the two or more sub-parts of the deduplication data structure whose number of allocated entries exceeds the number of their associated target entries, wherein the entries are one of least recently used and least frequently used.

10. The apparatus of claim 1, wherein the at least one processing means is further configured to perform the step of: in response to determining that the amount of unused space in the given physical space block exceeds a specified threshold, performing a background data copying operation to reduce the amount of unused space in the given physical space block.

11. The device of claim 10, wherein the storage system includes a plurality of virtual logical blocks that map logical data blocks to the physical space of the plurality of physical space blocks, and wherein a given physical space block is limited to mapping a specified number of the plurality of virtual logical blocks.

12. The device of claim 11, wherein the background data copying operation includes: Migrate the first virtual logical block of the plurality of virtual logical blocks currently mapped to the given physical space block to another physical space block of the plurality of physical space blocks; as well as The second virtual logical block among the plurality of virtual logical blocks is assigned to the given physical space block.

13. The device of claim 12, wherein the first virtual logical block maps to a first logical data block having a first compression ratio, and the second virtual logical block maps to a second logical data block having a second compression ratio.

14. The device of claim 13, wherein the second compression ratio is lower than the first compression ratio.

15. A computer program product comprising a non-transitory processor-readable storage medium storing program code of one or more software programs therein, wherein the program code, when executed by at least one processing device, causes the at least one processing device to perform the following steps: To maintain a storage system a deduplicated data structure including a target total number of data block identifiers, the deduplicated data structure includes two or more sub-parts associated with different compression ratio ranges, the two or more sub-parts of the deduplicated data structure having corresponding target number of entries, the sum of the target number of the two or more sub-parts of the deduplicated data structure being less than or equal to the total number of target entries of the deduplicated data structure, the two or more sub-parts of the deduplicated data structure including a first sub-part and at least a second sub-part, the first sub-part having a first target number of data block identifiers for data blocks with a compression ratio in a first compression ratio range, the second sub-part having a second target number of data block identifiers for data blocks with a compression ratio in a second compression ratio range, the second target number of entries being different from the first target number of entries, and the second compression ratio range being different from the first compression ratio range; For a given data block to be stored in the storage system, identify the given data block identifier and the given compression ratio; Determine whether the given data block identifier is in a given sub-part of the two or more sub-parts of the deduplication data structure that has a given compression ratio range including the given compression ratio; In response to determining that the given data block identifier of the given data block is not in the given sub-part of the deduplication data structure, (i) the given data block is written to a given physical space block among a plurality of physical space blocks of the storage system, the given physical space block being selected at least in part based on the given compression ratio of the given data block and the amount of unused space in the given physical space block, and (ii) whether to add the given data block identifier as an entry to the given sub-part of the deduplication data structure, based at least in part on whether the number of allocated entries in the given sub-part of the deduplication data structure is less than the given target number of entries in the given sub-part of the deduplication data structure; as well as In response to determining the given data block identifier in the given sub-part of the deduplication data structure, the deduplication reference count of the given data block identifier is incremented.

16. The computer program product of claim 15, wherein the deduplication data structure comprises a deduplication hash table, and wherein the data block identifier comprises a hash of the data block content.

17. The computer program product of claim 15, wherein the number of the given target entries of the given sub-part of the deduplication data structure is calculated at least in part based on the given compression ratio range associated with the given sub-part of the deduplication data structure.

18. A method, the method comprising: To maintain a storage system a deduplication data structure including a target total number of data block identifiers, the deduplication data structure includes two or more sub-parts associated with different compression ratio ranges, the two or more sub-parts of the deduplication data structure having corresponding target entry numbers, the sum of the target entry numbers of the two or more sub-parts of the deduplication data structure being less than or equal to the total number of target entries of the deduplication data structure, the two or more sub-parts of the deduplication data structure including a first sub-part and at least one second sub-part, the first sub-part having a first target entry number for data block identifiers of data blocks with a compression ratio in a first compression ratio range, the second sub-part having a second target entry number for data block identifiers of data blocks with a compression ratio in a second compression ratio range, the second target entry number being different from the first target entry number, and the second compression ratio range being different from the first compression ratio range; For a given data block to be stored in the storage system, identify the given data block identifier and the given compression ratio; Determine whether the given data block identifier is in a given sub-part of the two or more sub-parts of the deduplication data structure that has a given compression ratio range including the given compression ratio; In response to determining that the given data block identifier of the given data block is not in the given sub-part of the deduplication data structure, (i) the given data block is written to a given physical space block among a plurality of physical space blocks of the storage system, the given physical space block being selected at least in part based on the given compression ratio of the given data block and the amount of unused space in the given physical space block, and (ii) whether to add the given data block identifier as an entry to the given sub-part of the deduplication data structure, based at least in part on whether the number of allocated entries in the given sub-part of the deduplication data structure is less than the given target number of entries in the given sub-part of the deduplication data structure; as well as In response to determining the given data block identifier in the given sub-part of the deduplication data structure, increment the deduplication reference count of the given data block identifier; The method is performed by at least one processing device, the at least one processing device including a processor coupled to a memory.

19. The method of claim 18, wherein the deduplication data structure comprises a deduplication hash table, and wherein the data block identifier comprises a hash of the data block content.

20. The method of claim 18, wherein the number of the given target entries of the given sub-part of the deduplication data structure is calculated at least in part based on the given compression rate range associated with the given sub-part of the deduplication data structure.

Images