Improving available storage space in systems with varying data redundancy schemes

Abstract

A technique is configured to provide various data protection schemes, such as replication and erasure coding, for data blocks of volumes served by storage nodes of a cluster, and the cluster is configured to perform deduplication of the data blocks. Additionally, the technique is configured to ensure that each deduplicated data block meets the data redundancy guarantee of the data protection scheme while increasing the storage space of the storage node. In order to meet the data integrity guarantee while increasing the available storage space, the storage node performs periodic garbage collection for the data blocks according to the currently applicable data protection scheme to optimize storage.

Description

Technical Field

[0001] The present disclosure relates to protection of data served by storage nodes of a cluster, and more particularly, to protection of data while improving storage space efficiency of data served by storage nodes of a cluster. Background Art

[0002] A plurality of storage nodes organized as a cluster can provide a distributed storage architecture configured to serve storage requests issued by one or more clients of the cluster. The storage request is for data stored on a storage device coupled to one or more storage nodes of the cluster. The data served by the storage node can be distributed across a plurality of storage units embodied as persistent storage devices, such as hard disk drives, solid-state drives, flash memory systems, or other storage devices. The storage node can logically organize the data stored on the device into logical volumes (volumes) that can be accessed as logical units (LUNs). Each volume can be implemented as a set of data structures, such as data blocks storing data for the volume and metadata blocks describing the data of the volume. For example, metadata can describe, for example, a storage location for data on an identification device.

[0003] Specifically, a volume (such as a LUN) can be divided into data blocks. In order to support increased data durability, data blocks can be replicated between storage nodes. That is, in order to ensure data integrity (availability) in the event of a node failure, a redundancy scheme (such as duplication of blocks) can be adopted within the cluster. In order to improve storage efficiency, data redundancy methods other than duplication (such as erasure coding) can be used. Unlike data duplication in which no data is encoded and one or more copies of a data block can be obtained from a non-faulty node, some data is encoded using erasure coding and is used for reconstruction when a node failure occurs. However, supporting the same level of data integrity within a cluster for both replication redundancy and erasure-coded redundancy may be inefficient for storage space. This inefficiency may be exacerbated when the data redundancy levels between redundancy schemes are different. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The above and further advantages of the embodiments of the present invention may be better understood by referring to the following description in conjunction with the accompanying drawings, in which like reference numerals indicate identical or functionally similar elements, and in which:

[0005] Figure 1 is a block diagram of a plurality of storage nodes interconnected as a storage cluster;

[0006] Figure 2 is a block diagram of a storage node;

[0007] Figure 3A It is a block diagram of the storage service of the storage node;

[0008] Figure 3B is a block diagram of an exemplary embodiment of a storage service;

[0009] Figure 4 The diagram shows the write path of the storage node;

[0010] Figure 5 is a block diagram illustrating details of a block identifier;

[0011] Figure 6 illustrates an example workflow for a technique configured to provide multiple data protection schemes while maximizing available storage space;

[0012] Figure 7 illustrates an example workflow for a technique configured to provide multiple data protection schemes including erasure coding while maximizing available storage space; and

[0013] Figure 8 is a flow chart illustrating the operation of a method for garbage collection of data blocks in accordance with multiple data protection schemes. DETAILED DESCRIPTION

[0014] Overview

[0015] Embodiments described herein are directed to a technique configured to improve storage utilization for various data protection schemes (such as replication and erasure coding) for data blocks of a logical volume ("volume") served by a storage node of a cluster configured to perform deduplication of the data blocks. Additionally, the technique is configured to ensure that each deduplicated data block complies with the data integrity guarantees of the data protection scheme while reducing the storage space of the storage node. That is, for multiple data protection schemes applied to the data blocks, the storage of redundant information necessary to provide the data integrity guarantees can be reduced while maintaining the same data integrity guarantees.

[0016] Each volume can be implemented as a set of data structures, such as data blocks storing data for the volume and metadata blocks describing the data of the volume. Volumes can be divided into data blocks. The storage service implemented in each node includes: a metadata layer having one or more metadata (slice) services configured to process and store metadata; and a block server layer having one or more block services configured to process (deduplicate) data and store the data on the node's storage device. It should be noted that the block service is configured to provide the maximum degree of data protection proposed by various data protection schemes, and data blocks are still deduplicated across volumes despite various data protection schemes between volumes.

[0017] When data protection is provided in the form of replication (redundancy), the slice service of the storage node initially generates one or more copies of the data block to be stored on the cluster. According to the technology, depending on the data protection scheme, the volume is assigned to the slice service. For example, when triple replication protection of data is provided, the slice service generates three copies of the data block (i.e., original copy 0, copy 1, and copy 2) by synchronously copying (replicating) the data block to the persistent storage of the additional slice service of the storage node in the cluster. Then, by asynchronously refreshing the data block copy to the corresponding block service, the copy of the data block is essentially represented as a copy of the cluster (i.e., original copy 0, copy 1, and copy 2). Therefore, the blocks of the volume can be assigned to the original copy 0 (R0) block service, as well as the primary copy 1 (R1) block service and the auxiliary copy 2 (R2) block service. Each replicated data block is illustratively organized in a specified container maintained by the block service of each node to be stored on the storage device. Each container is assigned to one or more block services based on the maximum replication scheme. Illustratively, for a triple-replication data protection scheme, three chunk services are assigned to each container. The chunk service calculates the container number and consults the container assignment table to identify the storage node with the corresponding chunk service that wrote the data chunk.

[0018] According to the technology, data blocks are also associated (marked) with an indication of their corresponding protection scheme. For example, data blocks of a volume with dual replication data protection (i.e., each data block has one replica) may assign the data blocks to two block services because the R0 data blocks are assigned to the R0 block service, while the R1 data blocks are assigned to the same container but to a different block service, namely the primary R1 block service. Illustratively, a data block may belong to a first volume with dual replication data protection and a different second volume with triple replication data protection. The technology ensures that the data blocks have enough replicas to satisfy the volume with a higher (highest) data integrity guarantee (i.e., the highest data protection scheme). Illustratively, the node's slice service can then issue a storage request based on the replication scheme to asynchronously refresh a copy of the data block (e.g., for dual replication, copies R0, R1; or, for triple replication, copies R0-R2) to the block service associated with the identified storage node.

[0019] When data protection is provided in the form of erasure coding, the block service can select the data blocks to be erasure coded. The data block set can then be grouped together to form a write group for erasure coding. Illustratively, the write group membership is guided by a changing group of containers so that the data is resilient to failures, such as assignments based on changing a subset of bits in a container identifier. The slice service routes data blocks and replicas of different containers to its associated block service. The implementation varies with the EC scheme selected for deployment (e.g., 4 data blocks and 2 encoding blocks for correction, referred to as 4+2EC). The block service assigns data blocks to containers based on a cryptographic hash and groups several different containers together based on the EC scheme deployed, for example, in a 4+2EC scheme, 4 containers can be grouped together (i.e., 4 uncoded data blocks + 2 encoding blocks with correction information), while in an 8+1EC scheme, 8 containers can be grouped together. Write groups of blocks from different containers can be selected from data blocks temporarily spooled according to the container. That is, data chunks of different containers of a write group are selected (i.e., picked) on a container-by-container basis from a temporarily spooled pool of chunks to represent a broad selection of containers with different failure domains that are resilient to data loss. Note that only data chunks (i.e., unencoded chunks) need to be assigned to containers, whereas encoded chunks can be simply associated with a write group by referencing the data chunks of the write group.

[0020] In the example, consider a block with a first data protection scheme using double replication and a second data protection scheme using 4+1 erasure coding, so that each scheme has a single redundancy for the unavailability of any one block. The blocks can be grouped into sets of 4 blocks, and the erasure coding scheme can be applied to form coded blocks (e.g., parity blocks), thereby generating 5 blocks for each set of 4 blocks, instead of 4 blocks and 4 copies (i.e., 8 blocks in total) for the replication scheme. It should be noted that the technology allows the selection of protection schemes (e.g., 4+1EC or double replication) on a block-by-block basis based on a set of capable protection schemes that meet the same redundancy level for the block according to the policy. For example, a performance-oriented policy can select a double replication scheme in which an uncoded copy of the block is always available without the need for parity calculations. On the other hand, a storage space-oriented policy can select an EC scheme to eliminate duplicates, thereby using storage more efficiently. Illustratively, 4 copies from the above-mentioned double replication scheme and 5 blocks from the 4+1EC scheme (9 blocks in total) can be consumed to store 4 data blocks. Thus, to maintain single failure redundancy, four of the duplicate blocks may be eliminated, thereby reducing the storage space of the storage node while maintaining the same data integrity guarantees against a single failure.

[0021] In order to meet data integrity guarantees while improving available storage space, storage nodes can perform periodic garbage collection for data blocks according to the currently applicable data protection scheme to optimize storage. The slice service of the storage node manages the metadata of each volume in the slice file and generates a list or Bloom filter for each data protection scheme at garbage collection time. Bloom filters identify data blocks currently associated with data protection schemes, and block services use Bloom filters to determine whether the data protection scheme for any data blocks they manage may have changed. If the (multiple) applicable data protection schemes (i.e., a set of capable protection schemes) for the data block have changed, the block service can improve (e.g., reduce redundant information) the storage consumption of the data block (e.g., better performance, better storage space efficiency) according to the policy and the currently applicable scheme to maintain the data integrity level associated with the changed (i.e., current) block. That is, as the redundancy scheme is changed, the same redundancy level of the data associated with the current block can be maintained.

[0022] For example, a data block may be marked with a dual replication scheme and a triple replication scheme. In this way, the original copy and two copies of the data block (i.e., replica 0, replica 1, and replica 2) are stored to comply with the dual and triple protection schemes. If the triple replication scheme is no longer applicable to the data block (e.g., the triple replication tag is eliminated), the third copy of the data block (i.e., replica 2) may be removed, leaving only replicas 0 and 1 stored to comply with the remaining dual replication scheme associated with the data block. If the data protection scheme associated with the data block is further changed to an erasure coding scheme and a storage space efficiency strategy is selected, the data block may be included in a write group with single parity protection and the second copy of the data block (i.e., replica 1) may be removed, so that the data block has the same redundancy level as the dual replication scheme. However, if a performance strategy is selected, replica 1 may not be eliminated. It should be noted that the change in data protection scheme is selected from a set of capable protection schemes available to the block.

[0023] When applying storage space efficiency strategies, other redundant information savings may occur even when the redundancy level of the erasure code is greater than the redundancy level used for replication. For example, consider the above set of 4 blocks using double replication (8 blocks in total, 4 of which are redundant blocks) to prevent any single random failure. The same 4 blocks can be protected against any two random failures using a 4+2EC scheme (6 blocks in total, 2 parity blocks). Although the data integrity guarantee may increase (i.e., 2 blocks may fail), 2 fewer blocks may be required. Illustratively, 10 blocks are initially stored: 4 original data blocks, 4 replicas through the replication scheme, and 2 parity blocks through the EC scheme. Thereafter, 4 of the replicas can be eliminated, leaving 6 blocks for use by the EC scheme (i.e., 4 data blocks and 2 parity blocks), with a higher level of data integrity than the blocks used by the replication scheme alone. For larger erasure coding schemes, such as 8+3EC based on Reed-Solomon codes, the savings can be even more significant: the double replication scheme requires 16 blocks (8 data blocks + 8 redundancy blocks), while the 8+3EC (8 data blocks + 3 parity blocks) scheme uses only 11 blocks. Therefore, the number of blocks required to support 3 random failures is reduced by 7, while previously only 1 random failure was possible.

[0024] illustrate

[0025] Storage Cluster

[0026] Figure 1 is a block diagram of a plurality of storage nodes 200 interconnected as a storage cluster 100 and configured to provide storage services for information (i.e., data and metadata) organized and stored on storage devices of the cluster. The storage nodes 200 may be interconnected via a cluster switch 110 and include functional components that collaborate to provide a distributed, scaled-out storage architecture for the cluster 100. The components of each storage node 200 include hardware and software functionality that enables the node to connect to and serve one or more clients 120 via a computer network 130, and also enables the node to connect to a storage array 150 of storage devices, thereby providing storage services in accordance with the distributed storage architecture.

[0027] Each client 120 may be embodied as a general-purpose computer configured to interact with the storage node 200 according to a client / server model of information transfer. That is, the client 120 may request a service from the node 200, and the node may return the result of the service requested by the client by exchanging packets through the network 130. When accessing information on the storage node in the form of storage objects (such as files and directories), the client may issue packets including file-based access protocols, such as the Network File System (NFS) and the Common Internet File System (CIFS) protocols over the Transmission Control Protocol / Internet Protocol (TCP / IP). However, in one embodiment, when accessing information in the form of storage objects (such as logical units (LUNs)), the client 120 illustratively issues packets including block-based access protocols, such as the Small Computer System Interface (SCSI) protocol encapsulated over TCP (iSCSI) and SCSI encapsulated over FC (FCP).

[0028] Figure 2 is a block diagram of a storage node 200, which is illustratively embodied as a computer system having one or more processing units (processors) 210, a main memory 220, a non-volatile random access memory (NVRAM) 230, a network interface 240, one or more storage controllers 250, and a cluster interface 260 interconnected by a system bus 280. The network interface 240 may include one or more ports suitable for coupling the storage node 200 to (multiple) clients 120 via a computer network 130, which may include a point-to-point link, a wide area network, a virtual private network implemented over a public network (the Internet), or a shared local area network. The network interface 240 thus includes the mechanical, electrical, and signaling circuitry required to connect the storage node to the network 130, which may include an Ethernet or Fibre Channel (FC) network.

[0029] Main memory 220 may include memory locations addressable by processor 210 for storing software programs and data structures associated with the embodiments described herein. Processor 210 may also include processing elements and / or logic circuitry configured to execute software programs (such as one or more metadata services 320a-n and block services 610-660 of storage service 300) and manipulate data structures. Operating system 225, portions of which typically reside in memory 220 (in a kernel) and are executed by a processing element (e.g., processor 210), functionally organizes storage nodes, inter alia, by invoking operations supported by storage service 300 implemented by the node. Suitable operating systems 225 may include general-purpose operating systems (such as Series or Microsoft A series of operating systems) or an operating system with configurable functionality (such as a microkernel and an embedded kernel). However, in the embodiments described herein, the operating system is illustratively Operating System. It will be apparent to those skilled in the art that other processing and memory components, including various computer-readable media, may be used to store and execute program instructions related to the embodiments herein.

[0030] The storage controller 250 cooperates with the storage service 300 implemented on the storage node 200 to access the information requested by the client 120. The information is preferably stored on a storage device (such as an internal solid-state drive (SSD) 270) illustratively embodied as a flash memory device and an SSD of an external storage array 150 (i.e., an additional storage array attached to the node). In one embodiment, the flash memory device can be a block-oriented device (i.e., a drive accessed as a block) based on a NAND flash memory component, such as a single-level cell (SLC) flash memory, a multi-level cell (MLC) flash memory, or a triple-level cell (TLC) flash memory, although it will be appreciated by those skilled in the art that other block-oriented, non-volatile solid-state electronic devices (e.g., drives based on storage class memory components) may be advantageously used with the embodiments described herein. The storage controller 250 may include one or more ports having an I / O interface circuit system that is coupled to the SSD 270 via an I / O interconnect arrangement (such as a traditional serial attached SCSI (SAS) and serial ATA (SATA) topology).

[0031] The cluster interface 260 may include one or more ports suitable for coupling the storage node 200 to other nodes (multiple) of the cluster 100. In one embodiment, dual 10Gbps Ethernet ports may be used for inter-node communication, although it will be apparent to those skilled in the art that other types of protocols and interconnects may also be used within the embodiments described herein. The NVRAM 230 may include a backup battery or other built-in last-state retention capability (e.g., non-volatile semiconductor memory, such as storage class memory) capable of maintaining data in the face of failures to the storage node and cluster environment.

[0032] Storage Services

[0033] Figure 3Ais a block diagram of a storage service 300 implemented by each storage node 200 of a storage cluster 100. The storage service 300 is illustratively organized as one or more software modules or layers that cooperate with other functional components of the node 200 to provide a distributed storage architecture of the cluster 100. In one embodiment, the distributed storage architecture aggregates and virtualizes components (e.g., network, storage, and computing resources) to present an abstraction of a single storage system with a large storage pool (i.e., all storage, including the internal SSDs 270 and external storage arrays 150 of the nodes 200 for the entire cluster 100). In other words, the architecture consolidates the storage of the entire cluster to support storage of LUNs, each of which can be allocated into one or more logical volumes ("volumes") having a logical block size of 4096 bytes (4KB) or 512 bytes. Each volume can also be configured with attributes such as size (storage capacity) and performance settings (quality of service) and access controls, and can thereafter be accessed by clients (i.e., exported to clients) as a block storage pool, preferably via iSCSI and / or FCP. Storage capacity and performance may then be “scaled” by growing (adding) the network, memory, and computing resources of nodes 200 to the cluster 100 .

[0034] Each client 120 may issue a packet as an input / output (I / O) request, i.e., a storage request, to access data of a volume served by a storage node 200, wherein the storage request may include data to be stored on the volume (i.e., a write request) or data to be retrieved from the volume (i.e., a read request), and client addressing in the form of a logical block address (LBA) or index for the volume based on the logical block size and length of the volume. The client addressing may be embodied as metadata that is separate from the data within the distributed storage architecture, so that each node in the cluster may store metadata and data on different storage devices (e.g., data on SSD 270a-n and metadata on SSD 270x) in the storage devices coupled to the node. To this end, the storage service 300 implemented in each node 200 includes: a metadata layer 310 having one or more metadata services 320a-n configured to process and store metadata, for example, on SSD 270x; and a block server layer 330 having one or more block services 610-660 configured to process and store data, for example, on SSD 270a-n. For example, metadata services 320a-n map between client addressing (e.g., LBA indexes) used by clients to access data on a volume and block addressing (e.g., block identifiers) used by block services 610-660 to store and / or retrieve data on a volume (e.g., an SSD).

[0035] Figure 3Bis a block diagram of an alternative embodiment of a storage service 300. When issuing a storage request to a storage node, the client 120 typically connects to the volume exported by the node (e.g., via an index or LBA). To provide an efficient implementation, the metadata layer 310 may alternatively be organized into one or more volume services 350a-n, where each volume service 350 may perform the functionality of the metadata service 320, but at the granularity of the volume, i.e., process and store metadata for the volume. However, the metadata for a volume may be so large that a single volume service 350 cannot process and store; therefore, multiple slice services 360a-n may be associated with each volume service 350. Therefore, the metadata for a volume may be divided into slices, and the slices of metadata may be stored and processed on each slice service 360. In response to a storage request for a volume, the volume service 350 determines which slice service 360a-n contains the metadata for the volume and forwards the request to the appropriate service 360.

[0036] Figure 4 The write path 400 of the storage node 200 for storing data on a volume of the storage array 150 is illustrated. In one embodiment, an exemplary write request issued by the client 120 and received at the storage node 200 (e.g., the master node 200a) of the cluster 100 may have the following form:

[0037] Write(volume,LBA,data)

[0038] Where volume specifies the logical volume to be written, LBA is the logical block address to be written, and data is the logical block size of the data to be written. Illustratively, the data received by the slice service 360a of the storage node 200a is divided into 4KB blocks. In box 402, each 4KB data block is hashed using a conventional cryptographic hash function to generate a 128-bit (16B) hash value (recorded as a block identifier (ID) of the data block); illustratively, the block ID is used to address (locate) data on the internal SSD 270 and the external storage array 150. Therefore, the block ID is an identifier of the data block generated based on the content of the data block. Conventional cryptographic hash functions, such as the Skein algorithm, provide a satisfactory random bit distribution within the 16B hash value / block ID adopted by the technology. In box 404, the data block is compressed using a conventional compression algorithm such as LZW (Lambo-Liv-Vecchio), and in box 406a, the compressed data block is stored in NVRAM 230. Note that in one embodiment, the NVRAM 230 is embodied as a write cache. Each compressed data block is then synchronously copied to the NVRAM 230 of one or more additional storage nodes (e.g., the secondary storage node 200b) in the cluster 100 for data protection (block 406b). When the data block has been securely and durably stored in the NVRAM 230a, 230b of the plurality of storage nodes 200a, 200b of the cluster 100, an acknowledgement is returned to the client.

[0039] Figure 5 506. 507 is a block diagram illustrating the details of a block identifier. In one embodiment, content 502 for a data block is received by a storage service 300. As described above, the received data is divided into data blocks having content 502, which can be processed using a hash function 504 to determine a block identifier (ID) 506. That is, the data is divided into 4KB data blocks, and each data block is hashed to generate a 16B hash value recorded as a block ID 506 of the data block; illustratively, the block ID 506 is used to locate the data on one or more storage devices 270 of the storage array 150. The data is illustratively organized in containers maintained by block services 610-660 for storage on the storage devices. The container can be derived from the block ID by extracting a predefined number of bits from the block ID 506 to store the corresponding data block.

[0040] In one embodiment, a container may be divided into buckets or "sublists" by extending a predefined number of bits extracted from a block ID. For example, the container field 508 of the block ID may contain the first two (e.g., most significant) bytes (2B) of the block ID 506, which are used to generate a container number (identifier) ​​between 0 and 65535 (depending on the number of 16 bits used) that identifies the container. The container identifier may also be used to identify a specific block service 610-660 and an associated SSD270. The sublist field 510 may then contain the next byte (1B) of the block ID, which is used to generate a sublist identifier between 0 and 255 (depending on the number of 8 bits used) that identifies a sublist with a container. In the event of a storage node failure or crash, dividing the container into sublists is particularly helpful for network transmission (or synchronization) of data between block services. The number of bits used for the sublist identifier may be set to an initial value and then adjusted later as needed. Each block service 610-660 maintains a mapping between a block ID and the location of a data block on its associated storage device / SSD (i.e., a block service driver (BSD)).

[0041] Illustratively, the block ID (hash value) can be used to distribute data blocks among containers in an evenly balanced (distributed) arrangement according to the capacity of the SSDs, where the balanced arrangement is based on "coupling" between the SSDs, i.e., each node / SSD shares approximately the same number of containers with any other node / SSD that is not in the same failure domain (i.e., protection domain) of the cluster. This is advantageous for rebuilding data in the event of a failure (i.e., rebuild) so that all SSDs will perform approximately the same amount of work (e.g., reading / writing data) to achieve fast and efficient rebuilding by evenly distributing the work among all SSDs of the storage nodes of the cluster. In one embodiment, each block service maintains a mapping of block IDs to data block locations on storage devices (e.g., internal SSD 270 and external storage array 150) coupled to the node.

[0042] Illustratively, container assignments may be stored in a distributed key-value store across the cluster. Figure 4, the distributed key-value store may be embodied as, for example, a "Zookeeper" database 450 that is configured to provide a distributed, shared-nothing (i.e., no single point of contention and failure) database that is used to store container assignments (e.g., a container assignment table) and configuration information that is consistent across all nodes of the cluster. In one embodiment, one or more nodes 200c have a service / process associated with the Zookeeper database 450 that is configured to maintain container assignments (i.e., a map) in conjunction with a data structure (e.g., a container assignment table 470). Illustratively, the distributed Zookeeper is hosted on up to five (5) selected nodes in the cluster, with all other nodes connecting to one of the selected nodes to obtain container assignment information. Thus, these selected "Zookeeper" nodes have replicated Zookeeper database images distributed across different failure domains of the nodes in the cluster, such that there is no single point of failure for the Zookeeper database. In other words, other nodes issue Zookeeper requests to their nearest Zookeeper database image (Zookeeper node) to obtain current container assignments, which may then be cached at the node to improve access time.

[0043] For each data block received and stored in the NVRAM 230a, 230b, the slice service 360a, 360b calculates the corresponding container number and queries the container assignment table 470 to identify the SSD 270a, 270b to which the data block is written. In blocks 408a, 408b, the slice service 360a, 360b of the storage node 200a, 200b then issues a storage request to asynchronously flush a copy of the compressed data block to the block service (illustratively labeled 610, 620) associated with the identified SSD. An exemplary storage request issued by each slice service 360a, 360b and received at each block service 610, 620 may have the following form:

[0044] storage(block ID, compressed data)

[0045] The block service 610, 620 for each SSD 270a, 270b (or storage device of the external storage array 150) determines whether it has previously stored a copy of the data block. If not, the block service 610, 620 stores the compressed data block associated with the block ID on the SSD 270a, 270b. Note that the block storage pool of the aggregated SSDs is organized by the content of the block ID (rather than the time when the data was written or the source of the data), thereby providing a "content-addressable" distributed storage architecture for the cluster. In addition to at least two copies of each data block stored on at least two SSDs in the cluster, this content-addressable architecture facilitates "automatic" deduplication of data at the SSD level (i.e., "for free"). In other words, the distributed storage architecture utilizes a single copy of the data and online deduplication of other copies of the data, i.e., in the event of a hardware failure, there are at least two copies of the data for redundancy purposes.

[0046] Increase available storage space for changing protection schemes

[0047] Embodiments described herein are directed to a technique for improving storage utilization for various data protection schemes (such as replication and erasure coding) for data blocks of logical volumes served by storage nodes of a cluster, the cluster being configured to perform deduplication of the data blocks. In addition, the technique is configured to ensure that each deduplicated data block complies with the data integrity guarantees of the data protection scheme while increasing the storage space of the storage node. That is, for multiple data protection schemes applied to data blocks, the storage of redundant information necessary to provide the data integrity guarantees can be reduced while maintaining the same data integrity guarantees.

[0048] As described above, the storage service implemented in each node includes: a metadata layer having one or more metadata (slice) services configured to process and store metadata; and a block server layer having one or more block services configured to process (deduplicate) data and store the data on the node's storage device. It should be noted that the block service is configured to provide the maximum degree of data protection proposed by various data protection schemes, and despite the varying data protection schemes among the volumes, data blocks are still deduplicated across volumes.

[0049] When data protection is provided in the form of replication (redundancy), the slice service of the storage node generates one or more copies of the data block to be stored on the cluster. Illustratively, the slice service calculates a corresponding container number for the data block based on a cryptographic hash of the data block, and queries (i.e., looks up) a container assignment table to identify the storage node to store (i.e., write) the data block to. In this manner, the container assignment table tracks the copies of the data block within the cluster. The slice service of the additional node then issues a storage request to asynchronously refresh the copy of the data block to the block service associated with the identified storage node.

[0050] According to the technology, volumes are assigned to slice services depending on the data protection scheme. For example, when triple copy protection of data is provided, the slice service initially generates three copies of the data block (i.e., original copy 0, copy 1, and copy 2) by synchronously copying (replicating) the data block to the persistent storage (e.g., NVRAM) of the attached slice service of the storage node in the cluster. Then, by asynchronously refreshing the data block copy to the corresponding block service, the copy of the data block is essentially represented as a replica of the cluster (i.e., original replica 0, replica 1, and replica 2). Therefore, the blocks of the volume can be assigned to the original replica 0 (R0) block service, as well as the primary replica 1 (R1) block service and the secondary replica 2 (R2) block service. Each replicated data block is illustratively organized in a designated container maintained by the block service of each node for storage on a storage device. Each container is assigned to one or more block services based on a maximum replication scheme. Illustratively, for a triple replication data protection scheme, three block services are assigned to each container. Illustratively, each slice service calculates a corresponding container number for the data chunk and consults (eg, looks up using the container number as an index) the container assignment table 470 to identify the storage node to which the data chunk is written.

[0051] According to the technology, data blocks are also associated (marked) with an indication of their corresponding protection scheme. For example, data blocks of a volume with dual replication data protection (i.e., each data block has one replica) may assign the data blocks to two block services because the R0 data block is assigned to the R0 block service, while the R1 data block is assigned to the same container but to a different block service, namely the primary R1 block service. Illustratively, a data block may belong to a first volume with dual replication data protection and a different second volume with triple replication data protection. The technology ensures that the data block has enough replicas to satisfy the volume with the higher data integrity guarantee (i.e., the highest data protection scheme). Illustratively, the node's slice service can then issue a storage request based on the protection scheme to asynchronously refresh a copy of the data block (e.g., for dual replication, copies R0, R1; or, for triple replication, copies R0-R2) to the block service associated with the identified storage node.

[0052] When data protection is provided in the form of erasure coding, the block service can select data blocks to be erasure coded. When erasure coding is used, in addition to the data blocks, the storage node also uses erasure codes to algorithmically generate coded blocks. Typically, erasure code algorithms, such as Reed-Solomon, use n blocks of data to create additional k blocks (n+k), where k is the number of coded blocks of replication or "parity" used for data protection. Erasure coded data allows lost blocks to be reconstructed from any n blocks of the n+k blocks. For example, an 8+3 erasure coding scheme, i.e., n=8 and k=3, converts eight blocks of data into eleven data / parity blocks. In response to a read request, the data can then be reconstructed from any eight of these eleven blocks (if necessary). It should be noted that reads are preferably performed from eight uncoded data blocks, and reconstruction is used when one or more uncoded data blocks are unavailable.

[0053] The data block set can then be grouped together to form a write group for erasure coding. Illustratively, write group membership is guided by a changing container group so that the data is resilient to failures, such as assignments based on changing a subset of bits in a container identifier. The slice service routes data blocks and replicas of different containers (e.g., with different container groups) to their associated block services. The implementation varies with the EC scheme selected for deployment (e.g., 4 data blocks and 2 encoding blocks for correction, 4+2EC). The block service assigns data blocks to containers based on cryptographic hashes and groups several different containers together based on the deployed EC scheme, for example, in a 4+2EC scheme, 4 containers can be grouped together, and in an 8+1EC scheme, 8 containers can be grouped together. Write groups of blocks from different containers can be selected from data blocks temporarily spooled according to the container. That is, data blocks of different containers of the write group are selected according to the container from a temporarily spooled block pool to represent a wide selection of containers with different failure domains that are resilient to data loss. Note that only data chunks (ie, uncoded chunks) need to be assigned to a container, whereas coded chunks can be associated with a write group simply by referencing the data chunks of the write group.

[0054] In the example, consider that a block has a first data protection scheme using double replication and a second data protection scheme using 4+1 erasure coding, so that each scheme has a single redundancy for the unavailability of any one block. The blocks can be grouped into sets, 4 blocks constitute a set, and the erasure code scheme can be applied to form a coded block (e.g., a parity block), thereby generating 5 blocks for each set of 4 blocks, instead of 4 blocks and 4 copies (i.e., a total of 8 blocks) for the replication scheme. It should be noted that the technology allows the selection of a protection scheme (e.g., 4+1EC or double replication) on a block-by-block basis based on a set of capable protection schemes that meet the same redundancy level for the block according to the policy. For example, a performance-oriented policy can select a double replication scheme in which an uncoded copy of the block is always available without the need for parity calculations. On the other hand, a storage space-oriented policy can select an EC scheme to eliminate duplicates, thereby using storage more efficiently. Illustratively, 4 copies from the above-mentioned double replication scheme and 5 blocks from the 4+1EC scheme (a total of 9 blocks) can be consumed to store 4 data blocks. Thus, to maintain single failure redundancy, four of the duplicated blocks may be eliminated, thereby reducing the storage space of the storage node while maintaining the same data integrity guarantees against a single failure. In one embodiment, a policy may be selected by an administrator when a volume is created.

[0055] When applying storage space efficiency strategies, other redundant information savings may occur even when the redundancy level of the erasure code is greater than the redundancy level used for replication. For example, consider the above set of 4 blocks using double replication (8 blocks in total, 4 of which are redundant blocks) to prevent any single random failure. The same 4 blocks can be protected against any two random failures using a 4+2EC scheme (6 blocks in total, 2 parity blocks). Although the data integrity guarantee may increase (i.e., 2 blocks may fail), 2 fewer blocks may be required. Illustratively, 10 blocks are initially stored: 4 original data blocks, 4 replicas through the replication scheme, and 2 parity blocks through the EC scheme. Thereafter, 4 of the replicas can be eliminated, leaving 6 blocks for use by the EC scheme (i.e., 4 data blocks and 2 parity blocks), with a higher level of data integrity than the blocks used by the replication scheme alone. For larger erasure coding schemes, such as 8+3EC based on Reed-Solomon codes, the savings can be even more significant: the double replication scheme requires 16 blocks (8 data blocks + 8 redundancy blocks), while the 8+3EC (8 data blocks + 3 parity blocks) scheme uses only 11 blocks. Therefore, the number of blocks required to support 3 random failures is reduced by 7, while previously only 1 random failure was possible.

[0056] Illustratively, containers are assigned to container groups in a manner that optimizes the erasure coding process. For example, in the case of a triple-copy data protection scheme, in which three replica versions of each container (original replica 0, primary replica 1, and auxiliary replica 2) are generated, the containers in the container group are assigned so that the original replica 0 version of the container is assigned across multiple different block services, the primary replica 1 version of the container is assigned to a different block service, and the auxiliary replica 2 version of the container is assigned to another different block service. Data blocks can be stored in containers according to a replication-based data protection scheme until a sufficient number of blocks are available for the selected erasure coding deployment. One of the block services (primary replica block service) used as the primary replica in the different block services coordinates the erasure coding process and selects candidate data blocks from each container for erasure coding. The primary replica block service forms a write group with the data blocks that remain uncoded, and generates one or more coded parity blocks, for example, primary coded parity blocks and auxiliary coded parity blocks. As known to those skilled in the art, a variety of erasure coding algorithms can be adopted, such as Reed-Solomon. The coded parity blocks can be stored along with the block identifiers of each data block used to generate the coded block. Each replica block service updates its metadata mapping for the uncoded copy of the data block to point to (i.e., reference) the location of the coded data blocks (e.g., primary coded parity blocks and secondary coded parity blocks) on the storage device so that any read request for the data block can return the coded block. It should be noted that the coded blocks can be enumerated (i.e., primary, secondary, etc.) to support asymmetric erasure coding schemes. After storing and updating the mapping for the coded blocks, the primary replica block service can release the storage space occupied by the uncoded additional copies of the data blocks in the write group while maintaining replica 0.

[0057] In order to meet data integrity guarantees while increasing available storage space (i.e., reducing unnecessary storage of duplicate data blocks), storage nodes perform periodic garbage collection on data blocks according to the currently applicable data protection scheme to increase storage. The storage node's slice service manages the metadata for each volume in the slice file and generates a list or Bloom filter for each data protection scheme at garbage collection time. Bloom filters identify data blocks currently associated with a data protection scheme, and block services use Bloom filters to determine whether the data protection scheme for any data block they manage may have changed.

[0058] If the (multiple) applicable data protection schemes for a data block have changed, the block service optimizes (e.g., reduces redundant information) the storage of the data block according to the currently applicable scheme to maintain the data integrity level previously associated with the changed block. That is, when the redundancy scheme is changed, the same redundancy level of the data associated with the changed block is changed. For example, a data block may have been previously associated with both a dual replication scheme and a triple replication scheme. In order to comply with the triple replication scheme, the original copy and two copies of the data block (i.e., replica 0, replica 1, and replica 2) have been stored. If the triple replication scheme is no longer applicable to the data block, the third copy of the data block can be removed, leaving only replicas 0 and 1 stored to comply with the data integrity guarantees of the remaining dual replication scheme.

[0059] Figure 6 and Figure 7 An example workflow for providing multiple data protection schemes while increasing the available storage space for a cluster is illustrated. It should be noted that the workflow is annotated with a series of letters AG representing the stages of operation. Although the workflow (s) are ordered, these stages illustrate an example to help understand the present disclosure and should not be used to limit the claims. Subject matter falling within the scope of the claims may vary with respect to order and some operations. It should be noted that as storage nodes are added to the cluster (i.e., the cluster is expanded), data blocks and redundant information (e.g., parity blocks and replicas) can be rebalanced within the cluster because containers are reassigned in the expanded cluster.

[0060] Reference Figure 6In an example workflow 600, each storage node 200a-c includes a slice service 360a-c and a block service 610-630, respectively. Each block service 610-630 hosts container 1-0, container 1-1, and container 1-2, respectively, wherein each container is assigned to and managed by its corresponding block service. Prior to stage A, data blocks A and data blocks B are received from one or more clients 120 to be stored in storage nodes 200a-c. A write request from a client includes a data block and identifies a volume on which the data is to be stored. The slice service 360a of the storage node 200a serves as a management (original) slice service and processes requests from clients (i.e., a client-oriented slice service). The slice service 360a manages metadata in slice files 607a and slice files 607b (collectively referred to as slice files 607) that are copied to slice services 360b and 360c across storage nodes 200b and 200c. Illustratively, slice files have a one-to-one relationship (i.e., association) with volumes: slice file 607a stores metadata for volume 1, and slice file 607b stores metadata for volume 2. When a request is received for a particular volume, slice service 360a updates the corresponding slice file accordingly. For example, block B is received with a write request for volume 2, so slice service 360a updates slice file 607b accordingly by indicating that block B is associated with volume 2. Slice service 360a also notifies slice services 360b and 360c of the update to the slice file and provides metadata for the update.

[0061] In response to the write request for block B, the slice service 360a prepares block B for storage by the corresponding block service 610. As previously described, data blocks are stored by containers assigned to each block service according to the container assignment table 470. As noted, data blocks can be assigned to containers based on the leading bits of the block ID 506 for the data block (i.e., the leading bits of the container field 508). Also as previously described, the block ID can be generated based on a cryptographic hash of the data block, and the data block is then stored in a container corresponding to the container identifier of the container field 508. For example, assume that block B has a block ID with a leading bit of "1" in the container field 508, so it is assigned to container 1-0, and then container 1-0 is assigned to the block service 610. Note that as a result of deduplication, a single data block may be associated with multiple volumes. Illustratively, as indicated in the slice file 607, block A is associated with both volume 1 and volume 2, but is only stored once in container 1-0 in order to preserve storage space. In one embodiment, chunks are stored with chunk ID 506 so that the chunk service avoids storing duplicate copies of the chunk by determining that a chunk with the same hash identifier has already been stored.

[0062] According to this technology, a data block can have multiple applicable data protection schemes. The data protection scheme may include: Figure 6The double replication (replication) and triple replication shown, as well as Figure 7 Erasure coding or other techniques as shown may be used to achieve data durability. As indicated by slice file 607, the data protection scheme may be configured on a per-volume basis, e.g., volume 1 is configured with a triple-copy data protection scheme, while volume 2 is configured with a double-copy data protection scheme. A single data block (such as block A) belonging to multiple volumes may have varying data protection schemes and, therefore, varying data protection requirements. The techniques described herein ensure that each data block is protected by the highest level of protection scheme (i.e., the maximum data integrity guarantee with the highest fault tolerance) configured for any of the volumes using the data block. For example, data blocks belonging to volume 2 configured with a double-copy scheme and volume 1 configured with a triple-copy scheme are protected by the triple-copy scheme, which provides greater protection because it can tolerate two failures, not just one. As a result, as required by the triple-copy scheme for volume 1, block A is stored a total of three times, even though volume 2, which also includes block A, only requires two replications. Illustratively, however, according to the double protection scheme for volume 2, block B is only stored twice.

[0063] To support data protection schemes, replicas of containers are generated and assigned to block services across the cluster. In one embodiment, a container can contain data blocks from multiple volumes, where each container is also configured to support the highest level of data protection scheme available, even if the highest level of protection is not currently being utilized by any of the data blocks in the container. That is, containers are globally (i.e., across the cluster) assigned to block services based on data protection capabilities, even if no volumes or data are currently using the highest level of data protection. For example, in Figure 6 In , the highest level protection scheme is triple replication, so in addition to the container that hosts the replica 0 copy of the data block, two copies of each container are created and assigned to the block service. Container 1-0, illustratively maintained by block service 610, hosts the unencoded version / replica 0 copy of block A and block B. Container 1-1, illustratively maintained by block service 620, hosts the replica 1 (R1) copy of the data block as indicated by the "-1" of the "hosted replica" notation "container 1-1". Similarly, container 1-2, illustratively maintained by block service 630, hosts the replica 2 (R2) copy of the data block as indicated by the "-2" of the hosted replica notation "container 1-2". Specifically, container 1-1 hosts the R1 copy of block A and block B, and container 1-2 hosts the R2 copy of block A. Note that a container assignment service (not shown) may assign containers (i.e., create and modify container assignment table 470) such that containers are located across different failure domains, such as, for example, different SSDs, different storage nodes, and / or different node chassis.

[0064] Illustratively, at stage A, slice service 360a receives command 609 indicating that chunk A should be deleted from volume 1. Slice service 360a updates slice file 607a accordingly by removing the mapping between volume 1 and chunk A, as indicated by Figure 6 . The updated slice file 607a (i.e., metadata update) is replicated across the appropriate slice services. The slice service may delay the transmission of a command to the block service to delete the original copy or any copies of block A to avoid the performance impact of small random delete operations. Conversely, the input / output (I / O) performance of the block service (e.g., the I / O performance of SSD 270) may be improved by saving delete operations for scheduled garbage collection periods so that writes to the SSD can be linked.

[0065] Illustratively, in phase B, the slice service begins performing garbage collection to remove unused data blocks. Garbage collection may be performed periodically, such as once an hour or once a day. Prior to initiating the garbage collection process, the slice service may communicate using a consensus protocol that garbage collection should be performed. In one embodiment, the slice service may object to starting garbage collection when the current I / O request load exceeds a load threshold or exceeds a space threshold for the amount of storage space still available (i.e., garbage collection is less necessary). Conversely, when the amount of available storage space drops below a minimum space threshold, the slice service may request garbage collection outside of the scheduled regular garbage collection time. In one embodiment, as part of the garbage collection process, the slice service examines (i.e., traverses or spans) the slice file 607 to identify data blocks currently used by the cluster's volumes and generates a block list 608. Illustratively, the block list 608 includes a list of blocks for each data protection scheme employed, for example, a list for double-replicated blocks and a list for triple-replicated blocks. Alternatively, the block list 608 may contain other data structures, such as a Bloom filter or a range of block IDs, to indicate which blocks are in use so that unused blocks can be removed by garbage collection to improve storage efficiency. The block list 608 may also include a master list of blocks in use, regardless of the associated data protection scheme.

[0066] Illustratively, at stage A, block A is not mapped from volume 1, which is the only volume that needs to triple-replicate block A. Since the slice service traverses the slice file 607 during stage B, block A is not included in the list of triply replicated blocks and is only included in the list of doubly replicated blocks. After generating the block manifest 608, the slice service may publish the block manifest 608 to the chunk service. In one embodiment, the publishing by the slice service may be accomplished by sending a storage location for the block manifest 608 to the chunk service or notifying and providing the block manifest 608 to the chunk service to perform garbage collection.

[0067] In phase C, the block services continue the garbage collection process. Using the block list 608, each block service compares the block IDs of the data blocks stored in its assigned container with the block list 608 to determine whether the data blocks are still in use and therefore whether to be deleted. The block service checks the list based on the data protection scheme associated with the container currently being analyzed. For example, the block service 630 compares the block IDs of the data blocks stored in container 1-2 with the block IDs of the block list 608 for triply replicated blocks, because container 1-2 is a replica 2 of container 1 and therefore only contains triply replicated blocks. The block service 620 compares the block ID of block 1-1 with the double-replication list and the triply-replication list in the block list 608 to determine whether the block is still in use, because block 1-1 can host blocks for both data protection schemes.

[0068] Alternatively, in some embodiments, the slice service may generate the chunk manifest 608 such that all triply replicated chunks are also listed in the manifest for the doubly replicated chunks, since triply replicated chunks are also inherently doubly replicated. In such embodiments, the chunk service 620 may check the list only for doubly replicated chunks. Similarly, the chunk service 610 hosting the primary (original) copy of the data chunk may check both lists in the chunk manifest 608 (i.e., the doubly and triply replicated chunk lists), or may check the master list of chunks (not depicted). If the chunk ID for a stored data chunk is not found in the appropriate list in the chunk manifest 608, the chunk service may delete or free up storage space for the data chunk that is no longer in use.

[0069] Illustratively, as part of the garbage collection process of phase C, block service 630 deletes block A from replica 2 container 1-2 because block A no longer needs to be triply replicated. To increase available storage space, the technology is configured to store data blocks in an efficient manner that still satisfies any applicable data protection scheme. As a result of command 609 of phase A, block A is removed from volume 1, which requires a dual replication protection scheme, but is still included in volume 2, which is configured with a triple replication scheme. Since replica 2 copies of block A are not required to comply with the dual replication scheme, block A is removed from container 1-2, which provides a third replica. It is important to note that replica 0 and replica 1 copies of block A remain stored for use by volume 2 as required by its dual replication scheme. If block A is later deleted from volume 2 by client request, block A will no longer be used by any volume, and therefore, block A will be removed from the cluster by block service 610 and block service 620 in a later garbage collection.

[0070] As noted, in some embodiments, data blocks may be marked with their applicable data protection schemes. In one embodiment, each data block may be associated with a data protection bit mask, wherein an unasserted sentinel value (e.g., "0") indicates that the corresponding data protection scheme is not applicable, and an asserted sentinel value (e.g., "1") indicates that the corresponding data protection scheme is applicable. Illustratively, the first bit of the bit mask may correspond to double replication, the second bit to triple replication, the third bit to a 4+1 erasure coding scheme, and the fourth bit to a 4+2 erasure coding scheme. Thus, data blocks associated with double and triple replication will have a mask of, for example, "1100". Such a data protection bit mask may be maintained by the block service for each data block stored in its assigned container. For example, in stage C, after determining that block A is no longer required to be triple replicated, the block service 610 may update the bit mask associated with block A to reflect the change in the protection scheme. Similarly, the block service 620 may update the bit mask associated with replica 1 of block A.

[0071] In one embodiment, the slice service may replicate slice files 607 across storage nodes so that each slice service has information about all stored blocks and associated volume / data protection scheme mappings. Alternatively, the slice service may not replicate slice files 607, or may replicate the content / information in one or more slice files based only on the data protection scheme of the corresponding volume. For example, information related to volume 2 in slice file 607b may not be replicated to slice service 360c because volume 2 is only doubly replicated. As an additional example, each slice service may only store and manage slice file information for assigned volumes, for example, volume 1 may be assigned to slice service 360a and volume 2 may be assigned to slice service 360b. Here, each slice service may separately generate its own block manifest 608 based on the slice file information available to it, and transmit its unique block manifest to all block services across the storage nodes of the cluster. Thus, each block service may receive three versions of a doubly replicated block manifest and a triply replicated block manifest.

[0072] Reference Figure 7In the example workflow 700 of , block services 610-640 host containers 0-3, that is, the containers are assigned to and managed by their corresponding block services. As used herein, a container group identifies a container (container ID) from which data blocks are selected for erasure coding. The number of containers in a container group corresponds to the number of input data blocks of the erasure coding scheme; for example, a 4+2EC scheme (as described in workflow 700) uses four containers. Therefore, the container group assignment 705 specifies four container IDs: container 0, container 1, container 2, and container 3. Illustratively, the container group assignment 705 also specifies that the primary (master) replica block service 650 and the secondary replica block service 660 store replicas for each container. As indicated by the assignment notations "650:1" and "650:2", the primary replica block service 650 stores (hosts) replica 1 for each container in the container group, and the secondary replica block service 660 hosts replica 2 for each container in the container group. Container group assignments 705 may be generated by a master / manager of cluster 100 ("cluster master / manager") or other service (e.g., a not depicted container assignment service) that handles container assignments and is maintained in a container group assignment table (not shown) within zookeeper database 450. Note that the container assignment service ensures that (i) each original replica 0 version of a container selected for a container group is assigned to a different chunk service (e.g., containers 0-3 are assigned across chunk services 610-640), (ii) primary replica 1 versions of a container are assigned to the same chunk service (e.g., all replica 1s are assigned to primary chunk service 650), and (iii) secondary replica 2 versions of a container are assigned to the same chunk service (e.g., all replica 2s are assigned to secondary chunk service 660).

[0073] Illustratively, the container assignment service may also assign containers in a manner that enables containers to be located across different fault domains. For example, each container may be assigned to or may be selected from a different solid-state drive (SSD), a different storage node, and / or a different storage node chassis. In addition, the container assignment service may ensure that no block service hosts multiple copies of the same container. The container assignment service enables container group assignments 705 to be used separately for all block services, including primary replica block services 650 and secondary replica block services 660. As noted, block service 650 hosts the primary encoding replica and therefore serves as the primary replica block service 650 that coordinates the erasure coding process using container group assignments 605, while block service 660 hosts the secondary encoding replica and serves as the secondary replica block service 660.

[0074] Slice file 607 reflects volume and data protection scheme information for data blocks AD, which are assigned to the erasure coding data protection scheme configured on volume 1 and volume 2. However, block A is also configured to be protected by a triple replication scheme, as indicated in Figure 3. The technology described herein allows protection schemes to be prioritized according to policy, such as a replication-based data protection scheme taking precedence over an erasure coding scheme. In other words, if a data block has an applicable replication-based protection scheme, the data block may not be erasure coded. When reconstructing information in the event of a failure, that is, when the replica 0 copy of the data block is no longer available for reading, erasure coding may result in slower degraded reads. In this case, erasure coding requires rebuilding the lost data block, which takes more time than simply reading the uncoded copy of the lost data block (requiring reading other data blocks and parity to perform reconstruction). Therefore, the use of a replication-based redundancy scheme assumes that the slower degraded read of the EC scheme caused by the reconstruction may be an unacceptable performance impact on the data stored on the volume. As a result, block A may not be used for erasure coding because it is associated with the replication scheme, and before phase A, there are not enough blocks to satisfy the 4+2 erasure coding scheme.

[0075] In phase A, in response to receiving a command 709 indicating that block A should be deleted from volume 3, the slice service 360 ​​updates the slice file 3 of the slice file 607 to indicate that data block A is no longer associated with volume 3. Block A is therefore deleted from volume 3, and as a result, the data block is available for erasure coding. In one embodiment, erasure coding can be performed during a subsequent garbage collection or recycling period. The master replica block service 650 that coordinates the erasure coding process compares its block identifier with the block list based on the data protection scheme generated by the slice service. After updating block A to indicate that its only applicable data protection scheme is erasure coding, the block service 650 determines that there are available data blocks from each container in the container group for erasure coding, and performs the erasure coding process in phases B, C, and D. Performing erasure coding increases available storage space when possible because uncoded copies (i.e., replicas) for blocks AD can be removed. It should be noted that although there is still more than one replication scheme associated with the block, the recovery of the block can be determined based on the policy (e.g., erasure code recovery and replica recovery) as to which recovery scheme should be adopted.

[0076] In phase B, the primary replica block service 650 identifies data block AD as available for erasure coding and forms a write group with block AD. Block service 650 can analyze the data protection bit mask associated with each block that identifies the applicable data protection scheme to determine that the block can be erasure coded. In phase C, block service 650 generates coded block P and writes it to its own storage, and generates coded block Q and writes it to secondary replica block service 660. The primary replica block service 650 reads its uncoded copies of data blocks AD and processes these uncoded copies using an erasure coding algorithm to generate coded parity blocks P, Q. The primary replica block service 650 updates the metadata entries for data blocks A, B, C, D, such as corresponding mapping fragments, using mappings, which, in addition to the existing location mappings for the data blocks, also point to the coded parity block P on the BSD of the block service 650. The secondary replica block service 660 similarly updates its mappings for the data blocks to include the location of the coded parity block Q on the BSD of the block service 660. The coded parity block is also stored with the block ID of the data block in the write group, i.e., the data block AD used to generate the coded parity block. In stage D, the primary replica block service 650 deletes the uncoded copies of data blocks A, B, C, D from the block service 650, and the secondary replica block service 660 deletes the uncoded copies of data blocks A, B, C, D from the block service 660. Since the coded blocks P, Q have been created, the data blocks A, B, C, D are now protected by the 4+2 erasure coding scheme and can still be read (recovered) even after two failures. As a result, the uncoded copies of the data blocks can be deleted to free up storage space.

[0077] Figure 8 8 is a flow chart illustrating the operation of a method for garbage collection of data chunks in compliance with multiple data protection schemes in the storage service 300 (block 800). In one embodiment, the operation may be performed by the chunk services 610-660. Broadly speaking, the chunk service may identify, for each data chunk stored in one or more of its assigned containers, a currently applicable data protection scheme. If it is determined that the applicable data protection scheme for a data chunk has changed, the chunk service may reduce the storage overhead for the data chunk by deleting unnecessary copies or erasure coding the data chunk and other data chunks to allow for the release (i.e., increase) of storage space previously occupied by unencoded copies of the data chunk.

[0078] In box 802, the block service receives a data structure such as a Bloom filter that identifies the data blocks that are still in use for each applicable data protection scheme, and for each selected data block (box 804), determines the (multiple) data protection schemes currently assigned to the selected block (box 806). In box 808, the block service searches the corresponding data structure for the (multiple) data protection schemes using the identifier (block ID) of the selected data block. In decision box 810, a determination is made as to whether any of the (multiple) data protection schemes are no longer applicable to the selected data block. If not, the method returns to box 804; otherwise, in box 812, the storage of the selected data block is optimized (e.g., reduced) according to the changed data protection requirements. In decision box 814, a determination is made as to whether there are any additional data blocks to select, and if so, the method returns to box 804. Otherwise, the method ends in box 816.

[0079] Above Figures 6 to 8 The embodiments described in the foregoing illustrate some operations that can ensure maximization of available storage space without violating one or more applicable data protection schemes ("DPS"). Other situations not considered above, such as situations where erasure coded data blocks are associated with replication-based DPS, can be handled according to one or more operational rules that ensure that all applicable DPSs are satisfied.

[0080] Typically, garbage collection can be employed in a manner that does not violate the data integrity guarantees of any DPS. For example, if the same block is stored on a double-replicated volume and a triple-replicated volume, at least three copies of the block are required to exist. That is, data integrity guarantees a level of fault redundancy for the DPS (e.g., no data loss in the event of k failures). The slice service can specify the DPS of the data for all read and write operations. The specification of the DPS can allow the block service to perform additional error checking. Illustratively, the technology described herein gives priority to unencoded copies of stored data blocks over encoded parity copies of stored blocks; such prioritization provides customers with improved degraded read performance for data stored on volumes with replication-based DPS. In one embodiment, a write group is created for one DPS, which simplifies encoding, garbage collection, and container synchronization while reducing the DPS information required to store encoded blocks. In addition, maximizing storage space or performing deduplication between write groups with different numbers of encoded copies increases write amplification during garbage collection.

[0081] When executing a write request, a block service hosting an uncoded copy of a data block may write the uncoded copy if no existing uncoded copy of the block exists. A block service hosting an coded copy of a block may write the uncoded copy if no existing coded copy of the block matches the specified DPS of the write request, and no existing uncoded copy of the block exists. A block service hosting an uncoded copy of a data block may set the DPS bits of the (DPS) bit mask for the uncoded block if the bit mask for the uncoded block does not have the appropriate bits set. Similarly, a block service hosting an coded parity copy of a block may set the DPS bits for the uncoded block if the DPS bit mask for the uncoded block does not have the appropriate bits set and the coded block does not exist.

[0082] In one embodiment, when performing an erasure coding operation, the primary replica block service may encode uncoded data blocks when the replica count of a block according to the DPS is greater than the maximum replica count of all block-based replication schemes. That is, if there are an excessive number of replicas for a failure redundancy level associated with a volume, the excess number of uncoded data blocks may be redeployed as coded (i.e., parity) blocks instead of being marked as unavailable for eventual garbage collection.

[0083] Illustratively, various rules may apply to initiating a garbage collection process. A block service may begin garbage collection when it has received the final Bloom filter associated with a DPS from all slice services for all DPSs. If the DPS change for the volume has been completed for all encoded copies of a data block, the slice service may send the final Bloom filter to the block service for the unencoded copy of the block associated with the erasure coding-based DPS. Other rules may apply when determining whether a data block can be deleted or recycled. If a block is not used for the DPS of a write group, the block service may discard the block. If a block is not used by any DPS, or the maximum encoded / unencoded replica count for the DPS of all blocks is less than the current replica count, the block service may discard the unencoded parity block.

[0084] In addition, block services may follow certain rules during read operations. Illustratively, a block service hosting a copy of a data block may return an unencoded copy if an unencoded copy exists. A block service storing an encoded copy of a block may return an encoded copy if an unencoded copy of the block does not exist. However, an encoded copy may not be returned if the DPS of the encoded copy is incompatible with the DPS indicated in the read request. For example, if a 4+2 DPS scheme is specified in the read request, a 4+1 encoded copy may not be returned. An encoded copy may be returned if the DPS for the encoded copy has an encoded copy count greater than or equal to the applicable DPS specified in the read request (e.g., a k value for an erasure coding scheme, such as 3 in a 5+3 scheme).

[0085] The foregoing description has been directed to specific embodiments. However, it is apparent that other variations and modifications may be made to the described embodiments to obtain some or all of their advantages. For example, it is expressly contemplated that the components and / or elements described herein may be implemented as software encoded on a tangible (non-transient) computer-readable medium (e.g., a disk, an electronic memory, and / or a CD), which has program instructions executed on a computer, hardware, firmware, or a combination thereof. Therefore, this description is by way of example only and is not intended to limit the scope of the embodiments herein. Therefore, the purpose of the appended claims is to cover all such variations and modifications that fall within the spirit and scope of the embodiments herein.

Claims

1. A method for data protection, comprising: generating a data structure associating a data block with each of a plurality of data protection schemes supported by a distributed storage system having a cluster of storage nodes, wherein each of the plurality of data protection schemes is configurable on a per-volume basis and has data integrity guarantees; determining, using the data structure, that a first data protection scheme and a second data protection scheme are simultaneously employed for first data blocks of a first volume and a second volume of the cluster, respectively, wherein the first data protection scheme and the second data protection scheme have a first data integrity guarantee, and the first data protection scheme is replication and the second data protection scheme is a first erasure coding; as well as Based on the policy, apply one of the following: (1) reduce the storage of redundant information in the storage nodes of the cluster for the first data block according to the second data protection scheme, (2) maintain the storage of redundant information of the first data block to improve performance.

2. The method according to claim 1, wherein reducing the storage of redundant information for the first data block according to the second data protection scheme further comprises: One or more copies of the first data block are deleted.

3. The method of claim 2, wherein maintaining storage of redundant information to improve performance avoids parity calculation of the first erasure coding.

4. The method of claim 1 , wherein generating the data structure that associates a data block with each of a plurality of data protection schemes supported by the distributed storage system further comprises: A Bloom filter is generated for the associated data block to which the data protection scheme is applied.

5. The method of claim 1 , wherein generating the data structure associating a data block with each of the plurality of data protection schemes further comprises: It is determined which volumes utilize the first data block.

6. The method according to claim 1, further comprising: A data protection bit mask associated with the first data block is updated to indicate that one of the first data protection scheme and the second data protection scheme is removed from association with the first data block.

7. The method according to claim 1, further comprising: determining, using the data structure, that a third data protection scheme is associated with the first data block, wherein the third data protection scheme has a second data integrity guarantee that is greater than the first data integrity guarantee; wherein the third data protection scheme is a second erasure coding; as well as Based on the policy, the storage of redundant information for the first data block is reduced according to the third data protection scheme.

8. The method according to claim 1, further comprising: Redundant information from the first data protection scheme and the second data protection scheme is distributed across the nodes of the cluster according to a hash.

9. The method according to claim 1, further comprising: Expanding the cluster by adding storage nodes; The first data block and the redundant information for the first data block are rebalanced within the expanded cluster.

10. A system for data protection, comprising: A cluster of storage nodes, each respective storage node comprising a storage device coupled to a processor, wherein execution of instructions by the processor causes the cluster to: generating a data structure associating a data block with each of a plurality of data protection schemes supported by the storage node, wherein each of the plurality of data protection schemes is configurable on a per-volume basis, and With data integrity guarantee; determining, using the data structure, that a first data protection scheme and a second data protection scheme are simultaneously employed for first data blocks of a first volume and a second volume of the cluster, respectively, wherein the first data protection scheme has the same data integrity guarantee as the second data protection scheme, wherein the first data protection scheme is replication and the second data protection scheme is erasure coding; Based on the policy, do one of the following: (1) generating one or more coded blocks according to the erasure coding for a block group including the first data block, and marking unused copies of the block group that are not necessary to ensure the data integrity; (2) Maintaining storage of an unencoded copy of the first data block to improve performance and maintain the data integrity guarantee.

11. The system of claim 10, wherein the generating of one or more coded blocks according to the erasure coding of the block group further comprises reclaiming the unused replicas of the block group that are not necessary to maintain the data integrity guarantee.

12. The system of claim 10, wherein generating the data structure associating a data block with each of a plurality of data protection schemes supported by the storage node further comprises generating a Bloom filter for the data block to which the associated data protection scheme applies.

13. The system of claim 10, wherein the generating of the data structure associating a data block with each of the plurality of data protection schemes further comprises determining which volumes utilize the first data block.

14. The system of claim 10, wherein the instructions further cause the cluster to update a data protection bit mask associated with the first data block to indicate that the first data protection scheme is removed from association with the first data block.

15. The system of claim 10, further comprising: Additional storage nodes are added to expand the cluster, wherein the instructions are further configured to rebalance the first data blocks and the encoding blocks within the expanded cluster.

16. The system of claim 10, wherein the instructions further cause the cluster to: The first data block is stored in a first storage node of the cluster, the first storage node being selected based on a hash function applied to the first data block.

17. The system of claim 16, wherein the instructions further cause the cluster to: At least one of the coded blocks is stored in a second storage node of the cluster, the second storage node being selected based on the hash function applied to the at least one of the coded blocks.

18. A non-transitory computer readable medium containing executable program instructions which, when executed by a processor of a storage node in a cluster of a distributed storage system, cause the storage node to: generating a data structure associating a data block with each of a plurality of data protection schemes supported by the distributed storage system, wherein each of the plurality of data protection schemes is configurable on a per-volume basis and has data integrity guarantees; determining, using the data structure, that a first data protection scheme and a second data protection scheme are simultaneously employed for first data blocks of a first volume and a second volume of the cluster, respectively, wherein the first data protection scheme and the second data protection scheme have a first data integrity guarantee, and the first data protection scheme is replication and the second data protection scheme is a first erasure coding; as well as Based on the policy, apply one of the following: (1) reduce the storage of redundant information in the storage nodes of the cluster for the first data block according to the second data protection scheme, (2) maintain the storage of redundant information for the first data block to improve performance.

19. The non-transitory computer-readable medium of claim 18, wherein generating the data structure that associates a data block with each of a plurality of data protection schemes supported by the distributed storage system comprises: A Bloom filter is generated for the associated data block to which the data protection scheme is applied.

20. The non-transitory computer-readable medium of claim 18, wherein the executable program instructions further cause the storage node to update a data protection bit mask associated with the first data block to indicate that one of the first data protection scheme and the second data protection scheme is removed from association with the first data block.

Images