Block and snapshot deletion

Abstract

The present disclosure relates to block and snapshot deletion. In some examples, a system maintains mapping metadata for data in a storage device, the data being divided into blocks, the mapping metadata including block entries that map block identifiers of blocks to storage locations, wherein the block identifiers monotonically increase as the blocks are created. The system updates a time period in response to triggering snapshot creation of the data. In response to the deletion of a first block, the system updates a first block entry of the first block to include a creation time period identifier and a deletion time period identifier of the first block. In response to the deletion of a first snapshot of a plurality of snapshots, the system uses the creation time period identifier and the deletion time period identifier to determine whether to remove the first block entry from the mapping metadata.

Description

Technical Field

[0001] The present disclosure relates to the field of data storage. Background Art

[0002] A storage system may be used to store data. In some cases, the storage system may be an object-based storage system, where data is stored as objects. Objects may be divided into chunks. In some cases, deduplication may be applied to reduce or eliminate the occurrence of duplicate chunks stored in the storage system. Summary of the invention

[0003] According to an exemplary aspect of the present disclosure, a non-transitory machine-readable storage medium is provided, comprising instructions, which when executed cause a system to perform the following operations: maintaining mapping metadata of data in a storage device (storage), the data being divided into blocks, the mapping metadata comprising a block entry mapping a block identifier of the block to a storage location, wherein the block identifier increases monotonically with the creation of the block and the block is immutable; updating a time period in response to triggering the creation of a snapshot of the data, the snapshot comprising a time point representation of the data, wherein the time period has a range of block identifiers, wherein multiple snapshots share a reference to at least one block; updating a first block entry of the first block in response to the deletion of the first block to include a creation time period identifier and a deletion time period identifier of the first block, the first block entry being part of the block entry in the mapping metadata, the creation time period identifier identifying a time period when the first block was created, and the deletion time period identifier identifying a time period when the first block was deleted; and determining whether to remove the first block entry from the mapping metadata using the creation time period identifier and the deletion time period identifier in response to the deletion of the first snapshot among the multiple snapshots. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Some embodiments of the disclosure are described with respect to the following figures.

[0005] Figure 1 is a block diagram of an arrangement including a cluster of computing nodes and a shared storage pool according to some examples.

[0006] Figure 2 is a diagram illustrating creation of epochs when taking data snapshots according to some examples.

[0007] Figure 3 is a block diagram of a storage medium storing machine-readable instructions according to some examples.

[0008] Figure 4 is a block diagram of a system according to some examples.

[0009] Figure 5 is a flow diagram of a process according to some examples.

[0010] In all drawings, the same reference numerals denote similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the dimensions of some parts may be exaggerated to more clearly illustrate the examples shown. In addition, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0011] In the present disclosure, unless the context clearly indicates otherwise, the use of the terms "a / an" or "the" is intended to include the plural forms as well. Similarly, when used in the present disclosure, the terms "includes / including / comprises / comprising", "have / having" indicate the presence of stated elements, but do not exclude the presence or addition of other elements.

[0012] As used herein, the term "object" may refer to any identifiable data unit. For example, an object may be identified using an object identifier, an object name, a uniform resource identifier (URI), or any other type of identifier.

[0013] "Object store" refers to a data repository of objects. A data repository may refer to any logical or physical storage arrangement or data.

[0014] An object can be divided into a collection of chunks, where a collection of chunks can include a single chunk or multiple chunks. A "chunk" can refer to any segment of a larger data.

[0015] In some examples, the index may be used to map a chunk identifier to a storage location in the storage device that stores a corresponding chunk identified by the chunk identifier.

[0016] In some examples, the index may be in the form of a tree data structure such as a B-tree. A B-tree index may include block entries that map block identifiers to corresponding storage locations in a storage device where corresponding blocks are stored. Each block entry of the B-tree index may include a key-value pair; the key may be the block identifier, and the value is the corresponding storage location. In such examples, the B-tree index is in the form of a key-value store.

[0017] Although reference is made to a B-tree index as an example of an index that may be used to map a chunk identifier to a storage location, in other examples, other types of indexes may be used. More generally, mapping metadata that maps a chunk identifier to a storage location of a chunk may be employed.

[0018] A "chunk identifier" may refer to any value used to identify a chunk so that the chunk can be distinguished from another chunk.

[0019] In some examples of the present disclosure, as blocks are written to the storage system, the block identifier increases monotonically. In other words, when each new block (after the first block) is written to the storage system, the new block is assigned a corresponding block identifier, which has an incremental value (increased by a certain specified value, such as 1 or a different value) relative to the corresponding block identifier of the block written to the storage system immediately before the new block. In such an example, each consecutive block identifier assigned to the corresponding block written to the storage system is greater than each block identifier assigned to the previously written block. In such an example, the block written later may not be assigned a block identifier with a lower value than the previously written block. In this way, the values ​​of the consecutive block identifiers generated for the new block may not decrease relative to each other.

[0020] When incoming data is received to be written to the storage system, the index can be used to check to determine whether any blocks of the incoming data are already stored in the storage system. If so, as part of the deduplication process, the duplicate blocks of the incoming data are not stored again in the storage system. Instead, references to such duplicate blocks are stored, and a reference count may also be incremented to indicate the number of instances of each block that have been received.

[0021] In order to provide the ability to recover from data failures, data snapshots can be taken. A "snapshot" refers to a point-in-time copy of data. For example, a snapshot can be taken of a collection of volumes (one volume or multiple volumes). A "volume" can refer to a logical unit of data that can contain a collection of objects (e.g., one object or multiple objects, or (multiple) parts of (multiple) objects). A snapshot of a collection of volumes taken at time t contains the version of the data in the collection of volumes that existed at time t.

[0022] In some examples, a snapshot does not store data, but may include a reference to a block stored in a storage system. A "reference" to a block contained in a snapshot may include a block identifier for the block. Different snapshots may be taken at different points in time. When a failure or error is detected and the system performs a recovery process to recover the data that existed at a given point in time, the snapshot corresponding to the given point in time may be used to perform the recovery.

[0023] Snapshots taken at different points in time may share some references to blocks. For example, snapshot S1 taken at time t1 may include references to blocks C1, C2, and C3. If block C3 is deleted after time t1 but before time t2, snapshot S2 taken at time t2 may include references to blocks C1 and C2. In this example, snapshots S1 and S2 share references to blocks C1 and C2.

[0024] When an object is deleted in a storage system (e.g., due to a request from a user, program, or machine), the object's corresponding blocks may also be deleted. However, a problem associated with storage systems that employ snapshots is that snapshots may also reference blocks. As a result, even if blocks are deleted because the corresponding object is deleted, if the blocks are referenced by any snapshots, the deleted blocks are not removed from the storage system. In some examples, deleted blocks may be marked for removal by the storage system's garbage collection process. The "garbage collection" process may refer to the process of removing unused data to free up storage capacity of the storage system.

[0025] As used herein, deletion of a block refers to marking such a block for deletion. A deleted block (i.e., a block marked for deletion) is still stored in the storage system (in addition, metadata associated with such a deleted block may still be stored), where the metadata may be, for example, part of a B-tree index. Therefore, if a recovery operation is to be performed using a snapshot that references the deleted block, the block may still be available for performing the recovery operation.

[0026] Removing a deleted block refers to freeing the storage location occupied by the block, removing references to the block, etc., so that the block is no longer accessible within the storage system (e.g., via data structures of the storage system). Removal of blocks can be performed by a garbage collection process. Determining whether a deleted block can be removed during a garbage collection process may be inefficient in some cases because the garbage collection process may have to perform a potentially time-consuming search to determine whether any snapshots or other data containers reference the deleted block.

[0027] According to some embodiments of the present disclosure, in order to efficiently remove blocks, such as during a garbage collection process, epochs associated with snapshots are used. These epochs can track whether a deleted block is shared by any snapshot. These epochs can be used to more efficiently remove deleted blocks.

[0028] According to some examples of the present disclosure, when a data block is deleted, for the deleted block, the corresponding block entry of the metadata in the mapping metadata (e.g., B-tree index) is updated by adding a deletion marker (also referred to herein as a "tombstone").

[0029] In some examples, the deletion marker may include a deletion epoch identifier. Each block entry further includes a creation epoch identifier. When a snapshot is deleted, block entries with deletion markers (e.g., deletion epoch identifiers) are checked to see if they can be removed; the check determines whether the corresponding block is not shared by any surviving snapshots in the system (a "surviving snapshot" is a snapshot that has not been deleted or otherwise indicated as invalid).

[0030] An "epoch" represents a logical time associated with taking a snapshot. When a snapshot is taken, the current epoch rolls over to the next epoch. To this end, the epoch identifier increases monotonically as the epoch rolls over due to snapshot creation. Each epoch may include or be associated with epoch metadata, which includes epoch state information and ranges of chunk identifiers (discussed further below).

[0031] Techniques or mechanisms according to some embodiments of the present disclosure are applied in the context of snapshots referencing immutable blocks (ie, blocks can be deleted, but they cannot be overwritten by modified blocks).

[0032] 1. System layout

[0033] Figure 1 is a block diagram of an example system 100 including a cluster of computing nodes 102-1 through 102-N, where N ≥ 1. Figure 1 A plurality of computing nodes (eg, N>1) in the system 100 are shown, but in other examples only a single computing node (eg, N=1) may be provided in the system 100 .

[0034] Computing nodes 102-1 to 102-N are coupled to shared storage pool 104, for example, via a network. The network through which computing nodes can communicate with shared storage pool 104 may include a storage area network (SAN), a local area network (LAN), a wide area network (WAN), and the like.

[0035] The shared storage pool 104 includes a collection of storage devices (a single storage device or multiple storage devices). Each storage device may be implemented as a disk-based storage device, a solid-state drive, or any other suitable type of storage device.

[0036] In response to writes from compute nodes in 102 - 1 through 102 -N, shared storage pool 104 may store object store 106 containing various objects 108 .

[0037] In some examples, object store 106 may be in the form of a log structured file system (LFS), in which objects and associated metadata may be written to a log in sequence. In other examples, object store 106 may have other forms.

[0038] The shared storage pool 104 may also store a B-tree index 110 that contains mapping metadata for chunks of the objects 108 in the object storage 106. The B-tree index 110 has leaf nodes that contain chunk entries. The leaf nodes of the B-tree index 110 are nodes at the bottom of the hierarchy of the tree structure of the B-tree index (i.e., each leaf node does not have any child nodes). The leaf nodes may include a plurality of chunk entries, wherein each chunk entry maps a chunk identifier to a storage location in the object storage 106. The storage location may be in the form of an address, an offset, etc.

[0039] The B-tree index 110 has a root node at the top of the hierarchy of the tree structure of the B-tree index 110. The root node contains pointers to the child nodes of the root node, and each non-leaf node in the B-tree index 110 contains pointers to the child nodes of the non-leaf nodes. A "non-leaf node" refers to a node in the tree structure of the B-tree index 110 that is not a leaf node. The lowest level non-leaf node in the B-tree index 110 has a pointer to a leaf node in the B-tree index 110.

[0040] Although the examples refer to mapping metadata arranged in the form of a B-tree index 110, it should be noted that in other examples, the mapping metadata including the block entries may be in a different form (eg, a mapping table, a file, etc.).

[0041] According to some examples of the present disclosure, two different types of block entries may exist in the B-tree index 110, and more specifically, in the leaf nodes of the B-tree index 110. The first type of block entry is referred to as a "regular" block entry, such as the regular block entry 110-1. The second type of block entry in the B-tree index 110 is a tombstone entry, such as the tombstone entry 110-2.

[0042] Regular block entries are associated with blocks that are stored in object store 106 and have not been deleted. Logical deleted entries are associated with blocks that have been deleted but have not yet been removed from object store 106. Removal of deleted blocks may be performed by a garbage collector, such as any epoch-based garbage collector 112-1 through 112-N in respective compute nodes 102-1 through 102-N.

[0043] Removing a block may refer to removing the block from object storage 106 and removing metadata associated with the block from mapping metadata (eg, in the form of B-tree index 110 ) such that the block and associated metadata are no longer stored in storage pool 104 .

[0044] The differences between the regular block entry 110 - 1 and the tombstone entry 110 - 2 are discussed further below.

[0045] Snapshots 116 may also be taken of the data in object store 106. In some examples, object store 106 may include data volumes, where each volume may include a collection of objects 108. Each snapshot 116 may reference data of the collection of volumes.

[0046] The taking of snapshots may be controlled by snapshot controller (SC) 114 in computing node 102-1. In some examples where there are multiple computing nodes in system 100, one of the computing nodes may be designated or elected as a leader computing node. Any computing node in the cluster of computing nodes 102-1 to 102-N may be designated or elected as a leader computing node.

[0047] Leader compute node 102-1 may be responsible for taking snapshots of data in object store 106. In other examples, each of compute nodes 102-1 through 102-N (including non-leader compute nodes) may include a snapshot controller to take snapshots of data in object store 106.

[0048] The snapshots 116 taken by the snapshot controller 114 are taken at different points in time by the snapshot controller 114. Therefore, each snapshot 116 corresponds to a data version in the volume set at a corresponding point in time.

[0049] The computing nodes 102-1 to 102-N further include corresponding update buffers 118-1 to 118-N, which are stored in corresponding memories 126-1 to 126-N in the corresponding computing nodes 102-1 to 102-N. Each update buffer is a storage structure that stores data (including block entry updates) when a block is received. Although mentioned in the singular sense, an "update buffer" may also include multiple buffers.

[0050] The memory in the computing node can be implemented using a collection of memory devices (a single memory device or multiple memory devices). Examples of memory devices may include any one or some combination of the following: dynamic random access memory (DRAM) devices, static random access memory (SRAM) devices, flash memory devices, etc.

[0051] Each update buffer 118-i (i=1 to N) is used to store block entry updates associated with a block of incoming data to be written to object store 106. Incoming data refers to data to be written to object store 106. The incoming data may be associated with a write request issued by a requester (e.g., a user, a program, or a machine), which may be in communication with system 100 over a network, or alternatively, may be part of system 100.

[0052] The block entry update includes mapping metadata associated with the block to be written to object storage 106. The block entry update may include a block identifier that maps to a corresponding storage location in object storage 106. Figure 1 In the example of , update buffer 118-1 includes block entry updates 120-1 associated with a corresponding block to be written to object storage 106. Similarly, update buffer 118-N includes block entry updates. The block entry updates in each update buffer 118-i will be merged into the B-tree index 110 by the corresponding compute node 102-i.

[0053] As described above, each snapshot may be associated with a corresponding epoch. Creation of epochs is performed by epoch management logic 122, which in some examples may be part of snapshot controller 114. In other examples, epoch management logic 122 may be separate from snapshot controller 114.

[0054] In some examples, each epoch-based garbage collector 112-1 to 112-N and snapshot controller 114 can be implemented using one or more hardware processing circuits in a corresponding computing node, or a combination of machine-readable instructions executable on one or more hardware processing circuits in a corresponding computing node. The hardware processing circuit can include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit.

[0055] The epoch management logic 122 may create and update epoch information, including the epoch information 124 - 1 through 124 -N stored in respective memories 126 - 1 through 126 -N.

[0056] The period information 124-1 may include a set of period identifiers to identify corresponding periods. The period information 124-1 associates each identified period with corresponding period state information. In some examples, a period may have one of several different states: unprotected state, in progress state, protected state, and deleted state.

[0057] The period information 124-N includes similar information.

[0058] 2. Period

[0059] like Figure 2 As further shown in FIG. 1 , in response to a snapshot request 202 to take a snapshot (create a snapshot), the snapshot controller 114 may instruct each of the computing nodes 102-1 to 102-3 (in this example, assuming there are three computing nodes) in the cluster of computing nodes to prepare (quiesce) (204) the computing node instance of the object storage 106, freeze (206) the instance of the object storage 106, and then perform a snapshot (208).

[0060] When snapshot request 202 is received in time interval 220, epoch k (k ≥ 1) is the current epoch. Epoch k is associated with snapshot k to be created by snapshot request 202.

[0061] In time interval 220, the epoch state of epoch k is an unprotected state. In the unprotected state, epoch k is able to receive writes of blocks and updates of associated mapping metadata for blocks. An epoch receiving writes of blocks and updates of associated metadata refers to a condition in which object store 106 and B-tree index 110 are updated in response to data writes, and such updated data and associated metadata will be included in the next snapshot to be created. In other words, epoch k in the unprotected state can receive writes of blocks and updates of associated mapping metadata to be included in snapshot k to be created in response to snapshot request 202.

[0062] Quiescing an instance of object store 106 may refer to completing any in-progress writes at the corresponding compute node. Freezing an instance of object store 106 may refer to refusing to accept any write requests to object store 106. Taking a snapshot may refer to creating a portion of a snapshot by compute node 102-i.

[0063] As each compute node completes creation of its portion of the snapshot, the compute node may provide a completion response to snapshot controller 114. After receiving the completion response from the last compute node that responded to snapshot request 202, the current epoch (epoch k) rolls over (at 210) to the next epoch (epoch k+1).

[0064] In response to rolling from epoch k to epoch k+1, the epoch state of epoch k is transitioned from the unprotected state to the in-progress state, and the epoch state of epoch k+1 is set to the unprotected state.

[0065] In the ongoing state of epoch k, any blocks updated during time interval 220 (such as blocks in the write cache in compute node 102-i, not shown) and associated metadata (in update buffer 126-i) are synchronized by compute node 102-i to object storage 106 and B-tree index 110 in storage pool 104, respectively.

[0066] Once the roll is complete, the requested snapshot has been created and writes may now be resumed, as indicated by thawing ( 250 ) the instance of object store 106 .

[0067] Once the chunks and associated metadata have been synchronized to the storage pool 104 by the compute node 102-i, the compute node 102-i has committed its updates for epoch k to the persistent storage (in the storage pool 104), as indicated by the corresponding checkpoints 222-i. Checkpoint 222-1 indicates the time when the compute node 102-1 committed its updates for epoch k to the storage pool 104, checkpoint 222-2 indicates the time when the compute node 102-2 committed its updates for epoch k to the storage pool 104, and checkpoint 222-3 indicates the time when the compute node 102-3 committed its updates for epoch k to the storage pool 104.

[0068] The time when the last computation node submits its updates for epoch k is considered the point at which epoch k is committed, denoted as 224. Figure 2 In the example of , compute node 102 - 2 is the last node to submit its updates for epoch k, so the time of checkpoint 222 - 2 is the time when epoch k was submitted.

[0069] Snapshot k of epoch k refers to blocks committed by compute nodes 102-1 to 102-3 for epoch k. After epoch k is committed, the epoch state of epoch k is transitioned to a protected state.

[0070] In a time interval 226 immediately following the time interval 220, the period k+1 is in an unprotected state.

[0071] like Figure 2 As further shown in FIG. 2 , during time interval 226, snapshot controller 114 receives another snapshot request 230, which triggers a set of actions similar to those discussed above for epoch k. In response to snapshot request 230, epoch k+1 rolls over to epoch k+2 (at 232).

[0072] As described above, as chunks are written to the object store 106, the chunk identifier increases monotonically. In some examples of the present disclosure, chunks are immutable (i.e., chunks can be deleted, but cannot be overwritten by modified chunks). Due to the immutable nature of chunks, metadata for new chunks is added to the right side of the B-tree index 110, and removed chunks will not come back.

[0073] Furthermore, because the chunk identifier increases monotonically, the chunk entries added have successively increasing chunk identifiers as they are added to the right side of the B-tree index 110. Thus, in some examples, a later added chunk entry to the right of an earlier added chunk entry will have a larger chunk identifier than the earlier added chunk entry.

[0074] Due to this block identifier pattern added to the block entries of the B-tree index 110, the block identifiers can be divided into periods. Figure 1 , a range of chunk identifiers for epoch k is in segment 150-k of B-tree index 110, another range of chunk identifiers for epoch k+1 is in segment 150-k+1 of B-tree index 110, and yet another range of chunk identifiers for epoch k+2 is in segment 150-k+2 of B-tree index 110. A range of chunk identifiers for any epoch is distinct from and does not overlap a range of chunk identifiers for any other epoch.

[0075] The range of block identifiers for each epoch x can be defined by the maximum block identifier. Since the maximum block identifier of the previous epoch x-1 is known, the range of block identifiers for epoch x can be simply defined by one value (i.e., the maximum block identifier of epoch x). In such an example, the range of block identifiers for epoch x starts at the next block identifier immediately following the maximum block identifier of epoch x-1 and ends at the maximum block identifier of epoch x. In effect, the maximum block identifier identifies the boundaries of the ranges of block identifiers for different epochs.

[0076] In other examples, the range of chunk identifiers for each epoch x may be defined by a minimum chunk identifier and a maximum chunk identifier.

[0077] In B-tree index 110, in some examples, each regular block entry (e.g., Figure 1 110-1) may include the following information:

[0078] Chunk_ID→{Storage Location, Creation_epoch, Size, Checksum}.

[0079] “Chunk_ID” represents a chunk identifier of a chunk, “Storage Location” represents a storage location of the chunk in the object storage 106, “Creation_epoch” represents an epoch when a chunk entry is created in response to adding a corresponding chunk, “Size” represents a size of the chunk entry, and Checksum represents a checksum calculated based on the content in the chunk entry, wherein when the chunk entry is read, the checksum can be used to detect whether there is an error in the chunk entry.

[0080] When an existing block is deleted, the regular block entry of the existing block is converted into a logical deletion entry (e.g. Figure 1 110-2).

[0081] In some examples, a tombstone entry contains the following information.

[0082] Chunk_ID→{Storage Location, Creation_epoch, Deletion_epoch, Size, Checksum}.

[0083] In the logical deletion entry, "Deletion_epoch" is added, which is a deletion mark indicating that the block entry is a logical deletion entry. "Deletion_epoch" indicates the epoch when the corresponding block was deleted (or more precisely, the epoch when deletion was requested but the corresponding block has not been removed).

[0084] For example, if a given block is in epoch k (in Figure 2 220) added to the object store, Creation_epoch is set to epoch k (an identifier for epoch k). If a given block is later deleted in epoch k+n (where n≥1), Deletion_epoch is set to epoch k+n (an identifier for epoch k+n). Note that it is possible for a given block to be deleted in the same epoch in which it was added to the object store 106, in which case Creation_epoch and Deletion_epoch will have the same value.

[0085] A regular block entry for a given block is different from a logical delete entry for the given block because the regular block entry does not include a deletion period identifier. A block entry that does not include a deletion period identifier may refer to a block entry that omits an information element of the deletion period identifier, or alternatively, may refer to a block entry that does not have a valid value for the deletion period identifier.

[0086] 3. Garbage Collection

[0087] The garbage collection process may be triggered in response to the deletion of a snapshot. The deletion of a snapshot may be in response to a request from a user or another entity, in response to the application of a retention policy, etc. For example, a retention policy may specify that snapshots older than a specified time may be deleted.

[0088] According to some embodiments of the present disclosure, the garbage collection process may be performed by an epoch-based garbage collector 112 - i of each computing node 102 - i .

[0089] Each epoch-based garbage collector 112 - i may use the tombstone entries in the B-tree index 110 to identify snapshots that share a reference to a deleted block before removing the deleted block from the B-tree index 110 .

[0090] The epoch-based garbage collector 112 - i may determine which blocks with tombstone entries may be removed from the B-tree index 110 .

[0091] The inputs used by the epoch-based garbage collector 112-i are as follows: P, which represents the predecessor epoch (having a protected state or an in-progress state) immediately preceding the epoch associated with the deleted snapshot; and S, which represents the successor epoch (having a protected state or an in-progress state) immediately following the epoch associated with the deleted snapshot.

[0092] The epoch-based garbage collector 112-i traverses a subset of the B-tree index 110, starting at the block entry having a block identifier equal to the maximum block identifier of epoch P, and ending at the maximum block identifier of epoch r (where epoch r is associated with snapshot r being deleted). In this manner, the epoch-based garbage collector 112-i only checks block entries from the maximum block identifier of epoch r that is undergoing deletion to the maximum block identifier of epoch P (predecessor epoch), which reduces the search time in the B-tree index 110 for identifying block entries to be removed during garbage collection, thereby improving garbage collection efficiency.

[0093] In some examples of the present disclosure, the decision of whether to remove a logical deletion entry with Creation_epoch=C and Deletion_epoch=D is as follows:

[0094] IF(C>P)AND(D≤S): remove the logical deletion entry from the B-tree index 110;

[0095] ELSE: Skip and do not remove the tombstone entry.

[0096] The above criteria indicate that if the creation_epoch (C) of the logical delete entry is greater than the predecessor epoch (P), and the deletion_epoch (D) of the logical delete entry is less than or equal to the successor epoch, then the logical delete entry can be deleted. If the above criteria (C>P) AND (D≤S) are met, it means that no snapshots share the blocks associated with the logical delete entry except the deleted snapshot.

[0097] An example sequence of events is provided below.

[0098] Time T1 : Block 1000 is written to object store 106 at epoch k (ie, Creation_epoch=k).

[0099] After time T1: A series of epoch rollovers occur, advancing the epoch to the current epoch k+n.

[0100] Time T2: A delete request is issued for the block written at T1 with block identifier 1000. As a result of the delete request, the regular block entry for the block with block identifier 1000 is converted into a logical delete entry by adding a Deletion_epoch set to the current epoch k+n.

[0101] Time T3: Snapshot k associated with epoch k is deleted. In the example, the most recent predecessor epoch in the protected state or in-progress state is epoch k-1, and the least recent successor epoch in the protected state or in-progress state is epoch k+2. The most recent predecessor epoch of epoch k is the most recent epoch in the protected state or in-progress state before epoch k. The most recent successor epoch of epoch k is the earliest epoch in the protected state or in-progress state after epoch k. In this example, it is assumed that epoch k+1 is not in the protected state or in-progress state (e.g., epoch k+1 may have been deleted and is therefore in the deleted state).

[0102] In the example, assume that n>2, so that the current epoch is epoch k+3 or later. When checking the logical delete entry created at time T2, the garbage collector 112-i can determine that the creation epoch k of the block with block identifier 1000 is greater than the predecessor epoch k-1 (satisfying C>P), but the deletion epoch k+n is greater than the successor epoch k+2 (i.e., the condition D≤S is not satisfied). This indicates that the block is shared by all snapshots associated with the epochs starting at epoch k+2 and ending at epoch k+n-1. Therefore, the garbage collector 112-i does not remove the block with block identifier 1000 and the logical delete entry associated with the block with block identifier 1000 because other snapshots share the block with block identifier 1000.

[0103] In a different example, assume that n=2, so that the current epoch is epoch k+2. In this case, the garbage collector 112-i can determine that the creation epoch k of the block with block identifier 1000 is greater than the predecessor epoch k-1 (satisfying C>P), and the deletion epoch k+2 is equal to the successor epoch k+2 (i.e., satisfying the condition D≤S). This indicates that the block with block identifier 1000 is not referenced by any existing snapshot.

[0104] Typically, if C>P is not satisfied, this means that the creation period of the logical delete entry is less than or equal to P, which indicates that at least period P (and possibly other periods (multiple) earlier than period P) references the block with block identifier 1000 associated with the logical delete entry, making the block with block identifier 1000 unable to be removed.

[0105] If the condition D≤S is not satisfied, this means that the deletion epoch in the logical deletion entry is greater than S, which indicates that the block with block identifier 1000 associated with the logical deletion entry was deleted after epoch S and is therefore shared by a snapshot associated with an epoch starting at epoch S and ending at epoch k+n-1 (one less than the current epoch), making it impossible to remove the block with block identifier 1000.

[0106] 4. Further implementation methods

[0107] Figure 3 is a block diagram of a non-transitory machine-readable or computer-readable storage medium 300 storing machine-readable instructions that, when executed (eg, by one or more hardware processing circuits), cause the system to perform various tasks. The system may include one computer or multiple computers.

[0108] The machine-readable instructions include mapping metadata maintenance instructions 302 for maintaining mapping metadata (eg, Figure 1 The data is divided into blocks, and the mapping metadata includes a block entry that maps a block identifier of a block to a storage location (e.g., of the object store 106). The block identifier increases monotonically as blocks are created.

[0109] The machine-readable instructions include period update instructions 304 for updating the period in response to the creation of a snapshot of the triggering data. Updating the period may refer to changing the state of the period and rolling the period to the next period. In some examples, the period is associated with a corresponding snapshot, and the period has an identifier that increases monotonically as the period rolls over.

[0110] The machine-readable instructions include block entry update instructions 306 for updating a first block entry of the first block in response to the deletion of the first block to include a creation time identifier and a deletion time identifier of the first block, the first block entry being part of the block entry in the mapping metadata. The creation time identifier identifies the time when the first block was created, and the deletion time identifier identifies the time when the first block was deleted.

[0111] The machine-readable instructions include block entry removal instructions 308 for determining whether to remove the first block entry from the mapping metadata using a creation time identifier and a deletion time identifier in response to the deletion of the first snapshot in the plurality of snapshots. The creation time identifier and the deletion time identifier can be used to determine whether any snapshot references the first block. If any snapshot references the first block, then the removal of the first block is blocked. If no snapshot references the first block, then the removal of the first block is allowed.

[0112] In some examples, the mapping metadata is part of a tree structure index having a root node and leaf nodes, and the leaf nodes contain block entries. Removing the first block entry from the mapping metadata includes removing the first block entry from the leaf node of the tree structure index.

[0113] In some examples, the tree-structured index is divided into a plurality of segments associated with respective epochs, wherein each segment of the tree-structured index includes chunk entries having chunk identifiers within a respective chunk identifier range.

[0114] In some examples, using a creation period identifier and a deletion period identifier to determine whether to remove a first block entry from mapping metadata includes comparing the creation period identifier with a predecessor period identifier that identifies a predecessor period of a first period associated with a first snapshot being deleted, and comparing the deletion period identifier with a successor period identifier that identifies a successor period of the first period associated with the first snapshot being deleted.

[0115] In some examples, each of the first epoch and the predecessor epoch has a corresponding range of block identifiers. In response to the deletion of the first snapshot, the machine-readable instructions may traverse a subset of the mapping metadata starting from an end of the range of block identifiers of the predecessor epoch to an end of the range of block identifiers of the first epoch to identify the logically deleted entries to be removed. For increased efficiency, in response to the deletion of the first snapshot, the machine-readable instructions do not traverse other subsets of the mapping metadata to identify the logically deleted entries to be removed.

[0116] Figure 4 is a block diagram of a system 400 that includes a hardware processor 402 (or multiple hardware processors). The hardware processor may include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit.

[0117] System 400 includes storage media 404 storing machine-readable instructions executable on one or more hardware processors 402 to perform various tasks. Machine-readable instructions executable on a hardware processor may refer to instructions executable on a single hardware processor or instructions executable on multiple hardware processors.

[0118] The machine-readable instructions in the storage medium 404 include mapping metadata maintenance instructions 406 for maintaining mapping metadata of data in a storage device, the data being divided into blocks, the mapping metadata including block entries mapping block identifiers of blocks to storage locations, wherein the block identifiers monotonically increase as blocks are created.

[0119] The machine-readable instructions in the storage medium 404 include epoch-snapshot association instructions 408 for associating an epoch with a corresponding snapshot of the data, wherein each of the epochs includes epoch metadata including a corresponding chunk identifier range and an epoch state.

[0120] The machine-readable instructions in the storage medium 404 include period update instructions 410 for updating a current period in a period in response to triggering creation of a snapshot of the data, the snapshot comprising a point-in-time representation of the data, wherein updating the current period comprises triggering submission of data updates for the current period to a storage device and rolling over to a next period.

[0121] The machine-readable instructions in the storage medium 404 include block entry update instructions 412 for updating a first block entry of the first block to include a creation time identifier and a deletion time identifier of the first block in response to deletion of the first block, the first block entry being part of the block entry in the mapping metadata, the creation time identifier identifying a time when the first block was created, and the deletion time identifier identifying a time when the first block was deleted.

[0122] The machine-readable instructions in the storage medium 404 include block entry removal instructions 414 for determining whether to remove the first block entry from the mapping metadata using the creation time identifier and the deletion time identifier in response to the deletion of the first one of the snapshots.

[0123] Figure 5 is a flow diagram of process 500 according to some examples.

[0124] Process 500 includes maintaining (at 502) mapping metadata for data in a storage device, the data being divided into blocks, the mapping metadata including block entries mapping block identifiers of the blocks to storage locations, wherein the block identifiers monotonically increase as the blocks are created.

[0125] Process 500 includes updating (at 504) a current epoch in response to creation of a snapshot of triggering data, wherein the epoch has a block identifier range, wherein updating the current epoch includes rolling to a next epoch, and wherein each of the current epoch and the next epoch represents a respective logical time associated with taking of the snapshot.

[0126] Process 500 includes updating (at 506) a first block entry of the first block to include a creation time identifier and a deletion time identifier of the first block in response to deletion of the first block, the first block entry being part of the block entry in mapping metadata, the creation time identifier identifying a time when the first block was created, and the deletion time identifier identifying a time when the first block was deleted.

[0127] Process 500 includes performing tasks 508 and 510 in response to deletion of a first snapshot of the plurality of snapshots.

[0128] Task 508 includes using the creation epoch identifier and the deletion epoch identifier to determine whether any snapshots reference the first block.

[0129] Task 510 includes determining whether to remove a first block entry from the mapping metadata based on determining whether any snapshots reference the first block.

[0130] Storage media (e.g. Figure 3 300 or Figure 4 404) may include any one or some combination of the following: a semiconductor memory device, such as a dynamic or static random access memory (DRAM or SRAM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and a flash memory; a magnetic disk, such as a fixed disk, a floppy disk, and a removable disk; another magnetic medium, including a tape; an optical medium, such as a compact disk (CD) or a digital video disk (DVD); or another type of storage device. Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media distributed in a large system with possibly multiple nodes. Such one or more computer-readable or machine-readable storage media are considered to be part of an article (or product). An article or product may refer to any manufactured single component or multiple components. One or more storage media may be located in a machine running machine-readable instructions, or at a remote site, from which machine-readable instructions may be downloaded over a network for execution.

[0131] In the foregoing description, many details are set forth to enable an understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause a system to: maintaining mapping metadata for data in a storage device, the data being divided into blocks, the mapping metadata comprising block entries mapping block identifiers of the blocks to storage locations, wherein The chunk identifier increases monotonically as chunks are created, and the chunk is immutable; updating the epoch in response to triggering creation of a snapshot of the data, the snapshot comprising a point-in-time representation of the data, wherein a plurality of snapshots share a reference to at least one block; updating a first block entry of the first block to include a creation time identifier and a deletion time identifier of the first block in response to deletion of the first block, the first block entry being part of the block entry in the mapping metadata, the creation time identifier identifying a time when the first block was created, and the deletion time identifier identifying a time when the first block was deleted, wherein each time includes a block identifier range; and In response to deletion of a first snapshot of the plurality of snapshots, identifying block entries to be removed by traversing exclusively a subset of block entries having block identifiers in a traversal range based on corresponding creation and deletion epoch identifiers, wherein the first snapshot is associated with a first epoch, the first epoch is preceded by a predecessor epoch, and the traversal range begins after an end of a block identifier range of the predecessor epoch and ends at an end of the block identifier range of the first epoch.

2. The non-transitory machine-readable storage medium of claim 1, wherein: The mapping metadata is part of a tree structure index having a root node and leaf nodes, the leaf nodes containing the block entries, and the instructions include instructions for: using the creation and deletion epoch identifiers of the first chunk entry to determine whether to remove the first chunk entry from the mapping metadata, Wherein, removing the first block entry from the mapping metadata includes removing the first block entry from a leaf node of the tree structure index.

3. The non-transitory machine-readable storage medium of claim 2, wherein: The tree structure index is divided into a plurality of segments associated with respective epochs, each of the plurality of segments of the tree structure index including a chunk entry having a chunk identifier within a respective chunk identifier range.

4. The non-transitory machine-readable storage medium of claim 2, wherein: Using the creation and deletion epoch identifier of the first block entry to determine whether to remove the first block entry from the mapping metadata includes: comparing the creation epoch identifier of the first block entry to a predecessor epoch identifier identifying the predecessor epoch of the first epoch associated with the first snapshot being deleted, and The deletion epoch identifier of the first block entry is compared to a successor epoch identifier that identifies a successor epoch to the first epoch associated with the first snapshot being deleted.

5. The non-transitory machine-readable storage medium of claim 4, wherein: The preceding period is the most recent preceding period before the first period, ie, metadata of the period is being or has been submitted to the storage device.

6. The non-transitory machine-readable storage medium of claim 5, wherein: Comparing the creation time identifier to the predecessor time identifier includes determining whether the creation time identifier is greater than the predecessor time identifier.

7. The non-transitory machine-readable storage medium of claim 6, wherein: Determining to remove the first block entry from the mapping metadata is based on determining that the creation time identifier of the first block entry is greater than the look-ahead time identifier.

8. The non-transitory machine-readable storage medium of claim 7, wherein: Determining not to remove the first block entry from the mapping metadata is based on determining that the creation time identifier of the first block entry is not greater than the look-ahead time identifier.

9. The non-transitory machine-readable storage medium of claim 4, wherein: The subsequent epoch is the furthest subsequent epoch after the first epoch, the metadata of which is being or has been submitted to the storage device.

10. The non-transitory machine-readable storage medium of claim 9, wherein: Comparing the deletion epoch identifier to the successor epoch identifier includes determining whether the deletion epoch identifier is less than or equal to the successor epoch identifier.

11. The non-transitory machine-readable storage medium of claim 10, wherein: Determining to remove the first block entry from the mapping metadata is based on determining that the deletion epoch identifier is less than or equal to the successor epoch identifier.

12. The non-transitory machine-readable storage medium of claim 11, wherein: Determining not to remove the first block entry from the mapping metadata is based on determining that the deletion epoch identifier is not less than or equal to the successor epoch identifier.

13. The non-transitory machine-readable storage medium of claim 1, wherein: Updates to the period include: changing the state of the epoch from a state allowing data updates for the epoch to a state committing the data updates to the storage device, and Rollover from the stated period to the next period.

14. A system comprising: Hardware processor; as well as A non-transitory storage medium storing instructions executable on the hardware processor to: maintaining mapping metadata for data in a storage device, the data being divided into blocks, the mapping metadata comprising block entries mapping block identifiers of the blocks to storage locations, wherein the block identifiers monotonically increase as blocks are created; associating epochs with respective snapshots of the data, wherein each of the epochs comprises epoch metadata comprising a respective chunk identifier range and an epoch state; updating a current period in the periods in response to triggering creation of a snapshot of the data, the snapshot comprising a point-in-time representation of the data, wherein updating the current period comprises triggering committing data updates of the current period to the storage device and rolling over to a next period; updating a first block entry of the first block to include a creation time identifier and a deletion time identifier of the first block in response to deletion of the first block, the first block entry being part of the block entry in the mapping metadata, the creation time identifier identifying a time when the first block was created, and the deletion time identifier identifying a time when the first block was deleted; and In response to deletion of a first one of the snapshots, identifying block entries to be removed by traversing exclusively a subset of block entries having block identifiers in a traversal range based on corresponding creation and deletion epoch identifiers, wherein the first snapshot is associated with a first epoch, the first epoch is preceded by a predecessor epoch, and the traversal range begins after an end of the block identifier range of the predecessor epoch and ends at an end of the block identifier range of the first epoch.

15. The system of claim 14, wherein: Before updating the first block entry, the first block entry includes the creation period identifier but does not include the deletion period identifier.

16. The system of claim 14, wherein: The instructions include instructions for: in response to the deletion of the first snapshot, using the creation time identifier and the deletion time identifier to determine whether to remove the first block entry from the mapping metadata, including: comparing the creation period identifier with a predecessor period identifier, the predecessor period identifier identifying the predecessor period, and The deletion epoch identifier is compared to a successor epoch identifier that identifies a successor epoch to the first epoch associated with the first snapshot being deleted.

17. A method for a system including a hardware processor, the method comprising: maintaining mapping metadata for data in a storage device, the data being divided into blocks, the mapping metadata comprising block entries mapping block identifiers of the blocks to storage locations, wherein the block identifiers monotonically increase as blocks are created; updating a current epoch in response to triggering creation of a snapshot of the data, the snapshot comprising a point-in-time representation of the data, wherein the current epoch has a range of chunk identifiers, updating the current epoch comprises rolling to a next epoch, and each of the current epoch and the next epoch represents a respective logical time associated with taking of the snapshot; In response to deletion of a first block, updating a first block entry of the first block to include a creation time identifier and a deletion time identifier of the first block, the first block entry being part of the block entry in the mapping metadata, the creation time identifier identifying a time when the first block was created, and the deletion time identifier identifying a time when the first block was deleted; and In response to deletion of a first one of the snapshots, identifying block entries to be removed exclusively among a subset of block entries having block identifiers in a traversal range based on corresponding creation and deletion epoch identifiers, wherein the first snapshot is associated with a first epoch, the first epoch is preceded by a predecessor epoch, and the traversal range begins after an end of a range of block identifiers for the predecessor epoch and ends at an end of the range of block identifiers for the first epoch.

18. The method of claim 17, further comprising: In response to deleting the first of the snapshots: using the creation and deletion epoch identifiers of the first block entry to determine whether any snapshots reference the first block, and A determination is made whether to remove the first block entry from the mapping metadata based on a determination whether any snapshot references the first block.

19. The method of claim 17, wherein: Identifying block entries to be removed in response to deletion of the first snapshot includes exclusively traversing the subset of block entries having block identifiers in the traversal range based on corresponding creation and deletion epoch identifiers to identify block entries to be removed.

Images