Computing devices, methods, non-transitory machine-readable media for inline housekeeping of a deduplicated storage system

Abstract

Inline housekeeping of a deduplicated storage system is disclosed. Example embodiments relate to deduplication operations in a storage system. Examples include receiving a stream of data units to be stored in a persistent storage of a deduplicated storage system; loading a first container index into memory to index one or more data units included in the stream; updating the first container index to indicate a storage location of a first data unit included in the stream; reading a manifest list in the first container index to identify a first manifest indexed by the first container index; and decrementing a reference count of a second data unit in response to determining that the identified first manifest is included in a first backup item that has been marked for deletion while the first container index remains loaded into the memory to index the one or more data units.

Description

TECHNICAL FIELD

[0001] Computing devices, methods, non-transitory machine-readable media for inline housekeeping of a deduplicated storage system are disclosed. BACKGROUND

[0002] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. An example data reduction technique includes deduplication. Deduplication identifies duplicate data units and seeks to reduce or eliminate the number of instances of duplicate data units stored in a storage system. SUMMARY

[0003] A computing device comprising: a processor; a memory; and a machine-readable storage device having stored instructions executable by the processor to: receive a stream of data units to be stored in a persistent storage of a deduplicated storage system; load a first container index into the memory to perform data ingest of a set of received data units included in the stream, wherein the first container index remains loaded into the memory during the data ingest to deduplicate the set of received data units; update, during the data ingest of the set of received data units, the first container index loaded into the memory to indicate a storage location of a first data unit included in the set of received data units; read, during the data ingest of the set of received data units, a manifest list in the first container index loaded into the memory to identify a first manifest indexed by the first container index; determine, during the data ingest of the set of received data units, whether the identified first manifest is included in a first backup item that has been marked for deletion; and in response to determining that the identified first manifest is included in the first backup item that has been marked for deletion, decrement, during the data ingest of the set of received data units, a reference count of a second data unit identified in the first manifest in the first container index loaded into the memory. BRIEF DESCRIPTION OF DRAWINGS

[0004] Some embodiments are described with respect to the following figures.

[0005] Figure 1 is a schematic diagram of an example storage system in accordance with some embodiments.

[0006] Figure 2 is a diagram of an example data structure in accordance with some embodiments.

[0007] Figures 3A-3D is a diagram of an example process in accordance with some embodiments.

[0008] Figures 4A-4F is a diagram of example operations in accordance with some embodiments.

[0009] Figure 5 is a diagram of example processes in accordance with some embodiments.

[0010] Figure 6 is a diagram of an example computing device in accordance with some embodiments.

[0011] Figure 7 is a diagram of example processes in accordance with some embodiments.

[0012] Figure 8 is a diagram of an example machine-readable medium storing instructions in accordance with some embodiments.

[0013] In all of the drawings, like reference numerals refer to like parts throughout the several views. The drawings are not necessarily to scale, and the dimensions of some parts can have been exaggerated for the sake of clarity and convenience in illustrating the examples and / or embodiments presented herein. Furthermore, the drawings provide examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments provided in the drawings. DETAILED DESCRIPTION

[0014] In this disclosure, the use of the term “a,” “an,” or “the” is intended to mean one or more, unless the context clearly indicates otherwise. Likewise, the use of the term “includes,” “including,” “comprises,” “comprising,” “have,” or “having” is intended to mean that there are other items or elements, which are not recited, which are present or added, unless the context clearly indicates otherwise.

[0015] In some examples, a storage system can back up a collection of data (referred to herein as a “stream” or “data stream” of data) in deduplicated form, thereby reducing the amount of storage space required to store the data stream. The storage system can create a “backup item” to represent the data stream in deduplicated form. The storage system can perform a deduplication process, including breaking the data stream into discrete units of data (or “chunks”) and determining “fingerprints” of these incoming data units (as described below). Further, the storage system can compare the fingerprints of the incoming data units to the fingerprints of stored data units, and can thereby determine which of the incoming data units are duplicates of previously stored data units (e.g., when the fingerprints indicate a match). In the case of data units that are duplicates, the storage system can store a reference to the previously stored data unit, rather than storing the duplicate incoming data unit. The process for ingesting an incoming data stream and deduplicating it can be referred to herein as the “data ingestion” process of the storage system.

[0016] As used herein, a "fingerprint" refers to a value derived by applying a function to the content of a data unit (where "content" can include all or a subset of the content of the data unit). Examples of functions that can be applied include hash functions that produce a hash value based on the content of an incoming data unit. Examples of hash functions include cryptographic hash functions such as Secure Hash Algorithm 2 (SHA-2) hash functions (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions can be employed.

[0017] A "storage system" can include a storage device or an array of storage devices. A storage system can also include a storage controller(s) that manages access to the storage device(s). A "data unit" can refer to any portion of data that can be individually identified in a storage system. In some cases, a data unit can refer to a chunk, a collection of chunks, or any other portion of data. In some examples, a storage system can store data units in persistent storage. Persistent storage can be implemented using one or more persistent (e.g., non-volatile) storage devices such as disk-based storage device(s) (e.g., hard disk drive(s) (HDD(s)), solid state device(s) (SSD(s)) such as flash storage device(s), etc., or a combination thereof. A "controller" can refer to a hardware processing circuit, which can include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuit. Alternatively, a "controller" can refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) that are executable on the hardware processing circuit.

[0018] In some examples, a deduplicated storage system can use metadata to process an inbound data stream (e.g., a backup job). For example, such metadata can include a data recipe (also referred to herein as an "manifest") that specifies a receiving order of particular data units of each backup job. Further, such metadata can include job metadata to represent each received backup job (e.g., data stream) in a deduplicated form. The job metadata can include an identifier of a set of manifests and can indicate an order of the set of manifests. Processing each backup job can be referred to herein as a "backup process." Subsequently, in response to a read request, the deduplicated system can use the job metadata and the set of manifests to determine the receiving order of the data units, such that an original data stream of the backup job can be recreated. Accordingly, the set of manifests can be a representation of the original backup job. A manifest can include a series of records, each record representing a particular set of data unit(s). A record in a manifest can include one or more fields that identify a container index that indexes the data units (e.g., including storage information of the data units). For example, a container index can include one or more fields that specify location information (e.g., container, offset, etc.) of the stored data units, compression and / or encryption characteristics of the stored data units, etc. Further, a container index can include a reference count that indicates a number of manifests that reference each data unit.

[0019] In some examples, upon receiving a data unit (e.g., in a data stream), it can be matched against one or more container indexes to determine whether a same chunk is already stored in a container of the deduplicated storage system. For example, the deduplicated storage system can compare a fingerprint of the received data unit against fingerprints in the searched container index(s). If no matching fingerprint is found in the searched container index(s), the received data unit can be added to a container and an entry of the received data unit can be added to a container index corresponding to the container. However, if a matching fingerprint is found in the searched container index, it can be determined that a same data unit as the received data unit is already stored in the container. In response to the determination, a reference count of the corresponding entry is incremented and the received data unit is not stored in the container (as it already exists in one of the containers), thereby avoiding storing a duplicate data unit in the deduplicated storage system. As used herein, the term "matching operation" can refer to an operation to compare fingerprints of a set of data units (e.g., from a particular backup data stream) against fingerprints stored in a container index.

[0020] In some examples, a deduplicated storage system can perform housekeeping tasks to maintain the accuracy of included metadata. For example, when a data unit is deleted from a given manifest (e.g., due to a change to the data stream or item represented by the manifest), a housekeeping task can include decrementing the reference count for that data unit by one (i.e., indicating that one less manifest references the data unit). In some examples, when a particular backup item is marked for deletion in a deduplicated storage system (e.g., identified as no longer needed), a set of housekeeping tasks (referred to as a “housekeeping job”) can be scheduled and executed to delete the backup item. The housekeeping job can include identifying manifests included in the backup item, sequentially accessing and loading the manifests into memory, and reading the manifests to determine the container identifier and address information for data units in each manifest. The housekeeping job can also include accessing and loading the container index into memory (e.g., using the container identifier), decrementing the reference count for the data units (e.g., using the address information for the data units), and then saving the container index to persistent storage. Accordingly, performing such a housekeeping job can involve a relatively large amount of input / output (I / O) operations to load multiple metadata items (e.g., manifests and the container index) from persistent storage into memory, and to write at least some of the multiple metadata items from memory to persistent storage. Accordingly, performing such a housekeeping job can degrade the performance of the deduplicated storage system.

[0021] According to some embodiments of the present disclosure, a controller of a deduplicated storage system can perform at least some housekeeping tasks during a data ingest operation (also referred to herein as “inline housekeeping”). For example, when a container index is loaded into memory during a data ingest operation, the controller can access data structures in the container index to identify one or more manifests indexed by the container index. The controller can identify each backup item that includes at least one of the manifests indexed by the container index, and can determine whether the backup item has been marked for deletion. If so, the controller can perform an inline housekeeping operation to decrement the reference count in the container index representing data units included in the marked backup item. Accordingly, because the inline housekeeping operation for the marked backup item is performed while the container index is already loaded into memory (e.g., for the data ingest process), a separate I / O operation to load the container index into memory to perform housekeeping for the marked backup item is not needed. In this way, some embodiments can reduce the amount of I / O bandwidth used for housekeeping jobs, which can improve the performance of the storage system. Reference is made to the following detailed description of some embodiments of the present disclosure for further discussion of various aspects of the disclosed housekeeping process. Figures 1-8 Various aspects of the disclosed housekeeping process are further discussed.

[0022] Figure 1 - example storage system

[0023] Figure 1 An example of a storage system 100, including a storage controller 110, a memory 115, and a persistent storage device 140, is shown according to some embodiments. The persistent storage device 140 may include one or more non-transitory storage media, such as a hard disk drive (HDD), a solid-state drive (SSD), an optical disk, or a combination thereof. The memory 115 may be implemented using a semiconductor memory such as random access memory (RAM). In some examples, the storage controller 110 may be implemented via hardware (e.g., electronic circuitry) or a combination of hardware and programming (e.g., including at least one processor and instructions executable by said at least one processor and stored on at least one machine-readable storage medium).

[0024] like Figure 1 As shown, memory 115 and persistent storage device 140 can store various data structures, including at least project catalog 120, project metadata 130, list 150, container index 160, and data container 170. In some examples, copies of list 150, container index 160, and data container 170 can be transferred between memory 115 and persistent storage device 140 (e.g., via read and write input / output (I / O) operations).

[0025] In some implementations, storage system 100 may perform data ingestion operations to deduplicate received data. For example, storage controller 110 may receive an inbound data stream 105 comprising multiple data units (also referred to herein as a “backup item”) and may store at least one copy of each data unit in a data container 170 (e.g., by appending the data unit to the end of data container 170). In some examples, each data container 170 may be divided into entities 175, wherein each entity 175 comprises multiple stored data units.

[0026] In one or more embodiments, the storage controller 110 may generate a fingerprint for each received data unit. For example, the fingerprint may include a complete or partial hash value based on the data unit. To determine whether an incoming data unit is a copy of a stored data unit, the storage controller 110 may perform a matching operation to compare the fingerprint generated for the incoming data unit with a fingerprint in at least one container index 160. If a match is identified, the storage controller 110 may determine that the storage system 100 has already stored a copy of the incoming data unit. Instead of storing the duplicate incoming data unit, the storage controller 110 may then store a reference to the previous data unit.

[0027] In some embodiments, the storage controller 110 can generate item metadata 130 to represent each received backup item (e.g., data stream 105) in a deduplicated form. Each item metadata 130 can include an identifier of a set of manifests 150 and can indicate an order of the set of manifests 150. The manifests 150 record an order of received data units. Further, the manifests 150 can include pointers or other information that indicate a container index 160 that indexes each data unit. In some embodiments, the container index 160 can indicate a storage location of the data unit. For example, the container index 160 can include information that specifies a data unit is stored in an entity at a particular offset and the entity is stored in a data container 170 at a particular offset. Further, the container index 160 can include a reference count that indicates a number of manifests 150 that reference each data unit.

[0028] In some embodiments, the storage controller 110 can receive a read request to access stored data and, in response, can access the item metadata 130 and the manifests 150 to determine a sequence of data units that make up the original data. The storage controller 110 can then use the pointer data included in the manifests 150 to identify the container index 160 that indexes the data units. Further, the storage controller 110 can use the information included in the identified container index 160 (and the information included in the manifests 150) to determine a storage location of the data units (e.g., data container 170, entity 175, offset, etc.) and can then read the data units from the determined locations.

[0029] In some embodiments, the item directory 120 can be a data structure used to list backup items (e.g., backup items represented by item metadata 130) stored in the storage system 100 and a current status or state (e.g., active, deprecated, etc.) of each backup item. For example, the storage controller 110 can receive a request to delete a particular backup item and, in response, can update the item directory 120 to indicate that the backup item has a "deprecated" state (e.g., is pending deletion via a housekeeping process).

[0030] In some embodiments, performing the matching operation (e.g., during data ingestion) can include transferring a copy of the container index 160 from the persistent storage 140 to the memory 115 (e.g., via a read I / O operation). Upon completion of this transfer, the storage controller 110 can access the data structure in the container index 160 to identify the set of manifests 150 indexed by the container index 160. Further, the storage controller 110 can use the item directory 120 to determine whether any of the identified manifests include a backup item that has been marked for deletion (e.g., listed in the item directory 120 in a deprecated state). For example, a portion of each manifest identifier can be an item identifier of a backup item that includes the manifest 150. As such, the manifest identifier can be matched (via the item identifier) to a backup item that has been marked for deletion in the item directory 120. Further, upon identifying a manifest 150 that is indexed by the container index 160 and also included in a backup item that has been marked for deletion, the storage controller 110 can perform an inline housekeeping process to decrement the reference count in the container index 160 representing the data unit included in the backup item. Accordingly, because the inline housekeeping of the marked backup item is performed while the container index 160 remains loaded into the memory 115 (e.g., for the data ingestion process), a separate I / O operation to load the container index 160 into the memory 115 to perform housekeeping of the backup item is not required. In this way, some embodiments can reduce the amount of I / O bandwidth used for housekeeping jobs, which can improve the performance of the storage system 100. Reference is now made to Figures 3A-3D Further example processes for performing inline housekeeping are discussed.

[0031] Figure 2 - example data structures

[0032] Reference is now made to Figure 2 FIG. 1 shows a diagram of example data structures 200 used in deduplication, in accordance with some embodiments. As shown, the data structures 200 can include item metadata 202, manifests 203, a container index 220, and data containers 250. In some examples, the item metadata 202, manifests 203, container index 220, and data containers 250 can generally correspond to example embodiments of the item metadata 130, manifests 150, container index 160, and data containers 170 (as Figure 1 shown in FIG. 1), respectively. In some examples, the data structures 200 can be generated and / or managed by the storage controller 110 (as Figure 1 shown in FIG. 1).

[0033] In some embodiments, the item metadata 202 can include a plurality of manifest identifiers 205. Each manifest identifier 205 can identify a different manifest 203. In some embodiments, the manifest identifiers 205 can be arranged in a stream order (i.e., based on a receiving order of data units represented by the identified manifest 203). Further, the item metadata 202 can include a container list 204 associated with each manifest identifier 205. In some embodiments, the container list 204 can include an identifier of a set of container indexes 220 that index data units included in the associated manifest 203 (i.e., the manifest 203 identified by the associated manifest identifier 205).

[0034] Although only one of each data structure is shown for simplicity of illustration in Figure 2 , the data structure 200 can include multiple instances of the item metadata 202, each including or pointing to one or more manifests 203. In such examples, the data structure 200 can include multiple manifests 203. The manifests 203 can reference multiple container indexes 220, each corresponding to one of the multiple data containers 250. Each container index 220 can include one or more data unit records 230 and one or more entity records 240.

[0035] As shown in Figure 2 , in some examples, each manifest 203 can include one or more manifest records 210. Each manifest record 210 can include various fields, such as an offset, a length, a container index, and a unit address. In some embodiments, each container index 220 can include any number of data unit records 230 and entity records 240. Each data unit record 230 can include various fields, such as a fingerprint (e.g., a hash of the data unit), a unit address, an entity identifier, a unit offset (i.e., an offset of the data unit within the entity), a reference count value, and a unit length. In some examples, the reference count value can indicate a number of manifest records 210 that reference the data unit record 230. Further, each entity record 240 can include various fields, such as an entity identifier, an entity offset (i.e., an offset of the entity within the container), a storage length (i.e., a length of the data unit within the entity), a decompression length, a checksum value, and compression / encryption information (e.g., a compression type, an encryption type, etc.). In some embodiments, each container 250 can include any number of entities 260, and each entity 260 can include any number of stored data units.

[0036] In some embodiments, the unit address (included in the manifest record 210 and the data unit record 230) can be an identifier that deterministically identifies a particular data unit within a given container index 220. In some examples, the unit address can be a numerical value (referred to as an "arrival number") that indicates (e.g., when receiving an inbound data stream and de-duplicating it) an arrival order (also referred to as a "ingestion order") of the data unit being indexed in the given container index 220. For example, a first data unit being indexed in the container index 220 can be assigned an arrival number of "1" (e.g., by creating a new data unit record 230 for the first data unit), a second data unit can be assigned an arrival number of "2", a third data unit can be assigned an arrival number of "3", and so on. However, other embodiments are possible.

[0037] In some embodiments, the manifest record 210 can use a run-length reference format to represent a contiguous range of data units (e.g., a portion of a data stream) being indexed within a single container index 220. The run-length reference can be recorded in the unit address field and the length field of the manifest record 210. For example, the unit address field can indicate an arrival number of a first data unit in the represented range of data units, and the length field can indicate a number N of data units in the range of data units following the data unit specified by the arrival number in the unit address field (where "N" is an integer). The data units in the range of data units can have contiguous arrival numbers (e.g., because they are contiguous in the ingested data stream). As such, the range of data units can be represented by an arrival number of a first data unit in the range of data units (e.g., specified in the unit address field of the manifest record 210) and a number N of additional data units in the range of data units (e.g., specified in the length field of the manifest record 210). The additional data units in the range of data units following the first data unit can be deterministically obtained by calculating N arrival numbers sequentially following the specified arrival number of the first data unit, where the N arrival numbers identify the additional data units in the range of data units. In such examples, the manifest record 210 can include an arrival number "X" in the unit address field and a number N in the length field to indicate a range of data units including the data unit specified by the arrival number X and the data units specified by the arrival numbers X + i, where i = 0 to i = N (including 0 and N) (where "i" is an integer). In this way, the manifest record 210 can be used to identify all data units in the range of data units.

[0038] In one or more embodiments, the data structure 200 can be used to retrieve stored deduplicated data. For example, a read request can specify an offset and length of data in a given file. These request parameters can be matched with the offset and length fields of a particular manifest record 210. The container index and cell address of the particular manifest record 210 can then be matched with a particular data cell record 230 included in the container index 220. Further, the entity identifier of the particular data cell record 230 can be matched with the entity identifier of a particular entity record 240. Additionally, one or more other fields of the particular entity record 240 (e.g., entity offset, storage length, checksum, etc.) can be used to identify a container 250 and entity 260 from which the data cell can be read.

[0039] In some embodiments, each container index 220 can include a manifest list 222. The manifest list 222 can be a data structure for storing a set of entries, where each entry stores information about a different manifest 203 indexed by the container index 220. For example, in some embodiments, each entry of the manifest list 222 can store only a manifest identifier 205. Further, in other examples, each entry of the manifest list 222 can include both a manifest identifier 205 and at least one data cell range (e.g., a set of one or more data cells included in the manifest 203 and indexed by the container index 220). In some examples, a manifest list 222 having entries that include both a manifest identifier 205 and data cell range information can be referred to as a “reverse reference list.”

[0040] In some implementations, entries in Listing 222 may specify a data unit range as a run-length reference, which identifies the length of a first data unit and the data unit range in Container Index 220. For example, a run-length reference may identify a first data unit in the data unit range using a first arrival number “X” assigned to the first data unit in Container Index 220. Further, a run-length reference may identify a length value N indicating the number of arrival numbers sequentially following the first arrival number “X” and assigned to the remaining data units in the data unit range in Container Index 220. In such an example, a run-length reference included in a backreference entry may be used to identify the first data unit in the range (with a first arrival number “X”), and then the remaining data units in the range (with N arrival numbers consecutively following the first arrival number “X”). In this way, a run-length reference included in a backreference entry can be used to deterministically identify all data units in the data unit range. Further, in other examples, entries in Listing 222 may use other identifiers or representations to specify a data unit range (including one or more data units). For example, an entry can specify the range of data units as a fingerprint of a single data unit, a set of fingerprints of multiple data units, a set of arrival numbers of multiple data units, an offset (e.g., byte distance from the end), an offset, and a length, etc.

[0041] In some implementations, the storage controller (e.g., Figure 1 The storage controller 110 shown can load container index 220 into memory to perform a matching operation (e.g., during data ingestion of a received data stream). In response to (or subsequently) loading container index 220 into memory, the storage controller can access list 222 (included in container index 220) to identify list 203 indexed by container index 220. Further, the storage controller can determine whether any identified list 220 is included in a backup item that has been marked for deletion (e.g., in...). Figure 1 (As shown in the project catalog 120). If so, the storage controller can decrement the reference count in container index 220 (e.g., corresponding to a data unit included in a backup project) while container index 220 remains loaded into memory for matching operations. Accordingly, some implementations can reduce the amount of I / O bandwidth used for housekeeping jobs, thereby improving the performance of the deduplication storage system.

[0042] Figure 3A - An example procedure for initiating housekeeping

[0043] Figure 3AAn example procedure 300 for initiating housekeeping processes in a deduplication storage system, according to some embodiments, is shown. In some examples, a storage controller 110 (such as...) can be used. Figure 1 Process 300 is executed as shown. Process 300 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 300 can be executed by a single processing thread. In other embodiments, process 300 can be executed in parallel by multiple processing threads (e.g., simultaneously using a work diagram and performing multiple housekeeping tasks).

[0044] like Figure 3A As shown, box 310 may include monitoring events in the deduplication storage system. Decision box 312 may include determining whether data is being ingested by the deduplication storage system. If no (“No”), process 300 may continue at decision box 316 (as described below). Otherwise, if it is determined at decision box 312 that data is being ingested by the deduplication storage system (“Yes”), process 300 may continue at box 314, including performing inline housekeeping during data ingestion. See below for reference. Figures 3B-3C and Figures 4A-4D Some example extensions to description box 314 (i.e., example procedures for performing inline housekeeping).

[0045] Decision box 316 may include determining whether a housekeeping job has been scheduled in the deduplication store. If no (“No”), process 300 may return to box 310 (i.e., continue monitoring events in the deduplication store). Otherwise, if it is determined at decision box 316 that a housekeeping job has been scheduled in the deduplication store (“Yes”), process 300 may continue at box 318, including executing the scheduled housekeeping job. See below for reference. Figure 3D and 4E The example extension of box 318 describes a sample procedure for performing a scheduled housekeeping job. After box 318, procedure 300 may return to box 310 (i.e., continue monitoring events in the deduplication storage system).

[0046] In some embodiments, performing inline housekeeping during data ingest (e.g., at block 314) can complete some or all of the housekeeping work required by the deduplicated storage system. Accordingly, in such embodiments, the amount of housekeeping work to be performed by a scheduled housekeeping job (e.g., at block 318) can be eliminated or reduced (e.g., as compared to embodiments that lack inline housekeeping). For example, the scheduled housekeeping job can only need to perform housekeeping on a subset of the data units indexed by the container index that are no longer in use during data ingest (e.g., for matching operations against received data units). In this way, some embodiments can reduce the amount of I / O bandwidth used to perform housekeeping work, which can improve the performance of the deduplicated storage system.

[0047] Figure 3B and Figures 4A-4C Example procedure for inlining in-house processing

[0048] Figure 3B An example process 320 for inline housekeeping is shown in accordance with some embodiments. Process 320 can illustrate a first example extension of block 314 (as Figure 3A indicated) of process 300. Accordingly, in some embodiments, process 320 can be performed after determining that data is being ingested by the deduplicated storage system (e.g., from the “yes” output of decision block 312 as Figure 3A indicated).

[0049] For illustrative purposes, details of process 320 can be described below with reference to the example embodiments illustrated in Figure 1 and Figures 4A-4C However, other embodiments are possible. In some examples, process 320 can be performed using storage controller 110 (as Figure 1 indicated). Process 320 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by a processor(s)). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical, semiconductor, or magnetic storage medium. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 320 can be performed by a single processing thread. In other embodiments, process 320 can be performed by multiple processing threads in parallel.

[0050] As indicated in Figure 3A block 322 can include loading the container index into memory for matching against received data units during data ingest operations. Block 324 can include accessing the manifest list stored in the container index to identify a set of manifests indexed by the container index. For example, with reference to Figure 1 and Figures 4A-4BThe storage controller 110 initiates a data ingestion session for processing the inbound data stream 105. The storage controller 110 passes a copy of the container index 160 from the persistent storage 140 to the memory 115 (e.g., via a read I / O operation) for matching operations against the fingerprints of the received data units. Upon completion of this pass, the storage controller 110 reads the manifest list 445 (stored in the container index 160) to identify a set of manifests indexed by the container index 160.

[0051] Referring again to Figure 3B At block 326, a loop (defined by blocks 326-336) can be entered to process each manifest in the set of manifests identified by the manifest list in the container index. Block 328 can include determining a state of the backup item that includes the current manifest (i.e., the manifest being processed in the current loop iteration). Decision block 330 can include determining whether the backup item has a deprecated state. If no (“No”), the process 320 can return to block 326 (i.e., to process another manifest identified by the manifest list in the container index). For example, referring to Figure 4B The storage controller identifies the backup item that includes the set of manifests listed in the manifest list 445 (e.g., by parsing the identifiers of the manifests to extract the identifier of the backup item that includes the manifest). The storage controller then compares the identified backup item to the item catalog 450 and determines that the manifest “K-7” is included in the backup item “K” that has a “deprecated” state in the item catalog 450 (e.g., has been marked for deletion). In some examples, the item catalog 450 is a data structure that lists backup items stored in a deduplicated storage system (e.g., the storage system 100 shown in Figure 1 The storage controller 110 initiates a data ingestion session for processing the inbound data stream 105. The storage controller 110 passes a copy of the container index 160 from the persistent storage 140 to the memory 115 (e.g., via a read I / O operation) for matching operations against the fingerprints of the received data units. Upon completion of this pass, the storage controller 110 reads the manifest list 445 (stored in the container index 160) to identify a set of manifests indexed by the container index 160.

[0052] Referring again to Figure 3B If it is determined at decision block 330 that the backup item that includes the current manifest has a deprecated state (“Yes”), the process 320 can continue at block 331 including loading the current manifest into memory. Block 332 can include reading the loaded manifest to identify the data range(s) to decrement. Block 333 can include decrementing the reference count of the identified data unit range(s) in the container index. For example, referring to Figures 4B-4CThe storage controller loads a copy of the manifest "K-7" from the persistent storage 140 to the memory 115 (e.g., via a read I / O operation). The storage controller reads the loaded manifest "K-7" to identify one or more ranges of data units to decrement. The storage controller decrements the reference count of the identified ranges of data units (in the data unit metadata 442 of the container index 160). Further, the storage controller removes the entry for the manifest "K-7" from the manifest list 445.

[0053] Referring again to Figure 3B , the block 335 can include compressing the container index and saving the container index to persistent storage. For example, referring to Figure 2 and Figure 4A , the storage controller determines whether any reference counts in the container index 160 have been decremented to a zero value. If so, the storage controller attempts to obtain exclusive access to the container index 160 (e.g., if there are no matching operations to perform using the container index 160). Upon obtaining exclusive access to the container index 160, the storage controller compresses the container index 160, and then saves the container index 160 to the persistent storage 140.

[0054] Referring again to Figure 3B , the block 336 can include updating the item metadata to remove the container index from the list. For example, referring to Figure 2 and Figure 4A , the storage controller deletes the identifier of the container index 160 from the container list 204 associated with the current manifest 203 (i.e., the manifest being processed in the current loop iteration). Accordingly, the item metadata 202 is updated so that the container list 204 no longer identifies the container index 160 as indexing data units included in the current manifest 203. Further, if every container list 204 in the item metadata 202 is empty, the item metadata 202 can be deleted.

[0055] In some embodiments, the item metadata 202 can be saved to the persistent storage 140 when the data ingest session is complete. If changes to the item metadata 202 are lost (e.g., due to a power failure, system error, etc.) before the item metadata 202 is written to the persistent storage 140, the item metadata 202 in the persistent storage 140 can include incorrect information. For example, after recovering from a power failure, the container list 204 for the inventory "K-7" (in the item metadata 202) can still list the container index 160. However, the inventory list 445 in the container index 160 can no longer list the inventory "K-7." Upon identifying this discrepancy, the storage controller can determine that there is no current relationship between the inventory "K-7" and the container index 160, and can then remove the identifier for the container index 160 from the container list 204.

[0056] Referring again to Figure 3B , after block 336, the process 320 can return to block 326 (i.e., to process another inventory identified by the inventory list in the container index). Further, after all loop iterations are complete at block 326 (e.g., all inventories identified in the inventory list have been processed), the process 320 can complete.

[0057] Figure 3C and Figure 4D Example flow for inlining in-house processing

[0058] Figure 3C An example process 321 for in-line housekeeping using a reverse-reference data structure is shown, in accordance with some embodiments. The process 321 can illustrate a second example extension of the block 314 (as Figure 3A illustrated). Accordingly, in some embodiments, the process 321 can be performed after determining that data is being ingested by a deduplicated storage system (e.g., from the "yes" output of the decision block 312 illustrated). Figure 3A

[0059] For illustrative purposes, details of the process 321 can be described below with reference to the example embodiments illustrated in Figure 1 and Figures 4A-4D . However, other embodiments are possible. In some examples, the storage controller 110 (e.g., the deduplication module 114) can be used to perform the process 321. Figure 1 ​The process 321 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by a processor(s)). The machine- readable instructions can be stored in a non-transitory computer-readable medium such as an optical, semiconductor, or magnetic storage medium. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, and / or the like. In some embodiments, the process 321 can be executed by a single processing thread. In other embodiments, the process 321 can be executed by multiple processing threads in parallel.

[0060] In some embodiments, the process 321 can illustrate a modified version of the process 320 (as described above with reference to FIG. 3) in which the blocks 331, 332, and 333 (of the process 320) are replaced by the blocks 340 and 342 (of the process 321). Figure 3B The remaining blocks of the process 321 (i.e., the blocks other than 340 and 342) are described above with reference to the process 320. Figure 3B

[0061] The block 340 can include performing a lookup of the manifest in the reverse reference list of the container index to identify the data unit range(s). The block 342 can include decrementing the reference count of the data unit range(s) identified in the reverse reference list in the container index.

[0062] For example, with reference to FIG. 3, the block 340 can include performing a lookup of the manifest in the reverse reference list of the container index to identify the data unit range(s). The block 342 can include decrementing the reference count of the data unit range(s) identified in the reverse reference list in the container index. Figure 4D ​The storage controller reads a list of backreferences (“BR”) 448 (stored in container index 160) to identify a set of lists indexed by container index 160. The storage controller identifies backup items that include the set of lists listed in the backreference list 448 (e.g., by resolving the identifiers of the lists to extract the identifiers of the backup items that include those lists). The storage controller then compares the identified backup items with project directory 450 and determines that list “K-7” is included in backup item “K”, which has a “deprecated” status in project directory 450 (e.g., has been marked for deletion). The storage controller reads the entry for list “K-7” in the backreference list 448 to identify data unit range R1. The storage controller identifies the reference count (in data unit metadata 442 of container index 160) corresponding to the data unit included in data unit range R1 and then decrements the identified reference count for the data unit in data unit range R1. In some examples, the data range information included in the entries of the backreference list 448 may be specified as a run-length reference. In such an example, the storage controller reads the run-length reference (in the backreference entry) to identify the first data unit in the data unit range R1 (e.g., with a first arrival number "X"), and then identifies the remaining data units in the data unit range R1 (e.g., with N arrival numbers consecutively following the first arrival number "X"). In this way, the storage controller uses the run-length reference in the backreference entry to identify all data units in the data unit range R1, and then decrements the reference count corresponding to the data units identified in the data unit range R1 (in the data unit metadata 442 of container index 160). Accordingly, the backreference list 448 in container index 160 can be used to decrement the appropriate reference count for the housekeeping job without having to load list "K-7" into memory to identify the data units associated with those reference counts.

[0063] Figure 3D and Figures 4E-4F - An example process for performing a scheduled housekeeping job

[0064] Figure 3D An example process 350 for performing a scheduled housekeeping task is shown according to some embodiments. Process 350 can be illustrated by block 318 (e.g. Figure 3A (See example extension). Accordingly, in some implementations, process 350 may be executed after determining that a housekeeping job has already been scheduled in the deduplication storage system (e.g., from...). Figure 3A The "Yes" output of decision box 316 shown.

[0065] In some examples, storage controller 110 can be used ( Figure 1The process 350 can be performed by the storage controller 110 (shown in FIG. 1) in the manner shown. The process 350 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by processor(s)). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor memory device, or a magnetic memory device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, and the like. In some embodiments, the process 350 can be performed by a single processing thread. In other embodiments, the process 350 can be performed by multiple processing threads in parallel.

[0066] As shown in FIG. 3D, the block 352 can include scheduling a housekeeping job in a job queue. For example, referring to FIG. 4, the job queue 410 is a data structure that queues requests for deletion of backup items. In response to a received request, the storage controller schedules a plurality of housekeeping jobs in the job queue 410 to delete the backup items A-C. Further, the storage controller updates the item directory 450 to indicate that the backup items A-C have a "decommissioned" status (e.g., pending deletion in a housekeeping job). Figure 3D Figure 4E As shown in FIG. 3D, the block 352 can include scheduling a housekeeping job in a job queue. For example, referring to FIG. 4, the job queue 410 is a data structure that queues requests for deletion of backup items. In response to a received request, the storage controller schedules a plurality of housekeeping jobs in the job queue 410 to delete the backup items A-C. Further, the storage controller updates the item directory 450 to indicate that the backup items A-C have a "decommissioned" status (e.g., pending deletion in a housekeeping job).

[0067] As shown in FIG. 3D, the block 354 can include starting a housekeeping job to delete a backup item stored in the deduplicated storage system. The block 356 can include loading item metadata into memory. For example, referring to FIG. 4, the storage controller selects a housekeeping job from the job queue 410 and starts the selected job to delete the backup item A. Further, the storage controller loads the item A metadata 420 (i.e., the item metadata for the backup item A) from the persistent storage 140 into the memory 115. Figure 4E As shown in FIG. 3D, the block 360 can include reading a container list included in the item metadata. The block 364 can include generating a work entry in a work graph. For example, referring to FIG. 4, the storage controller reads the container list 430 included in the item metadata 420 for the backup item A. Further, the storage controller generates a work entry 460 in the work graph 440 to delete the container A1.

[0068] Figure 3D As shown in FIG. 3D, the block 360 can include reading a container list included in the item metadata. The block 364 can include generating a work entry in a work graph. For example, referring to FIG. 4, the storage controller reads the container list 430 included in the item metadata 420 for the backup item A. Further, the storage controller generates a work entry 460 in the work graph 440 to delete the container A1. Figure 4F ​​The storage controller reads item A metadata 420, which includes a plurality of manifest identifiers 422 and a plurality of container lists 425, where each container list 425 is associated with a different manifest identifier 422. Each container list 425 includes a different entry (also referred to as a "CI entry") for each container index 160 that indexes data units included in a manifest 150 identified by the associated manifest identifier 422. Further, for each listed container index 160, each container list 425 includes an ingest count value ("ingest count") that indicates a number of data units newly added to the container index 160 (i.e., that do not match the container index) from the manifest 150 at the time the identifier of the container index 160 was added to the container list 425. The storage controller uses the CI entries in the container lists 425 to add new work entries to the work graph 430.

[0069] Referring again to Figure 3D , block 368 can include processing a work entry in the work graph to complete the housekeeping job. For example, the storage controller selects a work entry for a container index (e.g., C-2) from the work graph and causes the container index to be loaded from persistent storage into memory. The storage controller reads a manifest list in the container index to identify a subset of manifests that match the backup item with the retired status. The storage controller causes the subset of manifests (i.e., the manifests in the manifest list that match the backup item with the retired status) to be loaded from persistent storage into memory. The storage controller reads the subset of manifests and identifies one or more data unit ranges to decrement. The storage controller decrements the reference counts for the identified data unit ranges in the container index.

[0070] Referring again to Figure 3D , decision block 370 can include determining whether the job queue is empty (e.g., no housekeeping jobs remain scheduled for execution). If not ("No"), the process 350 can return to block 354 (i.e., initiate another housekeeping job scheduled in the job queue for execution). Otherwise, if the job queue is determined to be empty at decision block 370 ("Yes"), the process 350 can complete.

[0071] Figure 5 - Example process for generating metadata

[0072] Figure 5 An example process 500 for generating metadata is shown in accordance with some embodiments. For illustrative purposes, details of the process 500 can be described below with reference to the example Figure 1 and Figures 4A-4F described above with reference to FIGS. 1-4. However, other embodiments are possible. In some examples, the storage controller 110 (e.g., the metadata generator 114) can perform the process 500. Figure 1The process 500 can be performed by the storage controller 110 (shown in FIG. 1) to store a backup item in a deduplicated storage system. The process 500 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by a processor(s)). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical, semiconductor, or magnetic storage medium. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.

[0073] Block 510 can include receiving a backup item to be stored in a persistent storage of a deduplicated storage system. Block 520 can include generating a fingerprint for a data unit of the received backup item. For example, referring to Figure 1 The storage controller 110 receives a backup item (e.g., the data stream 105) to be stored in the deduplicated storage system 100 and generates a fingerprint for a data unit in the received backup item.

[0074] Block 530 can include matching the generated fingerprints with fingerprints stored in a container index of the deduplicated storage system. Block 540 can include identifying a subset of the generated fingerprints that do not match the fingerprints stored in the container index. Block 550 can include storing the identified subset of the generated fingerprints in a set of container indexes. For example, referring to Figure 1 The storage controller 110 compares the generated fingerprints with the fingerprints included in the container index 160. If a match is identified for a data unit, the storage controller 110 determines that the storage system 100 has already stored a copy of the data unit. In response to the determination, the storage controller 110 stores a reference to a previous data unit (e.g., in the manifest 150) in a deduplicated form. Otherwise, if no match is identified for a data unit, the storage controller 110 stores the data unit in a data container 170 and adds an entry for the data unit to the container index 160 corresponding to the data container 170.

[0075] Block 560 can include generating a set of manifests to record an order of data units of the received backup item. Block 570 can include storing a reverse reference manifest in each container index to record a manifest and a data unit range that reference the container index. For example, referring to Figure 1 and Figure 4DThe storage controller 110 records the order in which data units are received in one or more manifests 150. Further, the storage controller 110 generates a reverse reference list 448 in the container index 160. The reverse reference list 448 identifies each manifest 150 (e.g., K-7) that currently references the container index 160. Further, the reverse reference list 448 can also identify, for each identified manifest 150, at least one data unit range that represents a set of data units included in the manifest 150 and indexed by the container index 160.

[0076] Block 580 can include generating project metadata to record the identifiers of the set of manifests. Block 590 can include storing at least one container list in the project metadata to record the set of container indexes and associated reverse reference flags. After block 590, the process 500 can complete. For example, referring to Figure 1 and Figures 4D-4F The storage controller 110 generates the project A metadata 420 to represent the backup project A. The project A metadata 420 includes a plurality of manifest identifiers 422 and a plurality of container lists 425, where each container list 425 is associated with a different manifest identifier 422. Each container list 425 includes a set of CI entries. Each CI entry includes a CI identifier of each container index 160 that indexes data units included in a manifest 150 identified by the associated manifest identifier 422. Further, each CI entry can include a reverse reference flag (not shown) to indicate whether the manifest-CI combination (represented by the CI entry) is to be processed using the reverse reference list 448 included in the container index 160.

[0077] Figure 6 - example computing device

[0078] Figure 6 A schematic diagram of an example computing device 600 is shown. In some examples, the computing device 600 can correspond generally to some or all of the storage system 100 (as shown in Figure 1 As shown, the computing device 600 can include a hardware processor 602, a memory 604, and a machine-readable storage 605 including instructions 610-650. The machine-readable storage 605 can be a non-transitory medium. The instructions 610-650 can be executed by the hardware processor 602 or by a processing engine included in the hardware processor 602.

[0079] The instructions 610 can be executed to receive a stream of data units to be stored in a persistent storage of a deduplicated storage system. For example, referring to Figure 1 and Figures 4A-4BThe storage controller 110 receives a backup item (e.g., the data stream 105) to be stored in the deduplicated storage system 100 and generates a fingerprint for a data unit in the received backup item.

[0080] The instructions 620 can be executed to load the first container index into memory to index one or more data units included in the stream. The instructions 630 can be executed to update the first container index loaded into memory to indicate a storage location of a first data unit included in the stream. For example, with reference to Figure 1 and Figures 4A-4B The storage controller 110 transfers a copy of the container index 160 from the persistent storage 140 to the memory 115 (e.g., via a read I / O operation) and performs a matching operation to compare the fingerprints of the received data units to the fingerprints included in the container index 160. The storage controller 110 identifies a first data unit having a fingerprint that does not match a fingerprint in the container index 160 and, in response, causes a copy of the first data unit to be stored in the data container 170. Further, the storage controller 110 updates the container index 160 to indicate a storage location of the first data unit in the data container 170.

[0081] The instructions 640 can be executed to read the manifest list in the first container index loaded into memory to identify a first manifest indexed by the first container index. For example, with reference to Figure 1 and Figures 4A-4B In transferring the container index 160 from the persistent storage 140 to the memory 115 to perform the matching operation, the storage controller 110 reads the manifest list (stored in the container index 160) to identify a set of manifests indexed by the container index 160.

[0082] The instructions 650 can be executed to decrement a reference count of a second data unit in the first container index in response to determining that the identified first manifest is included in a first backup item that has been marked for deletion, where the second data unit is identified in the first manifest, while the first container index remains loaded into memory to index one or more data units included in the stream. For example, with reference to Figures 4A-4CWhile the container index 160 remains loaded into the memory 115 for matching operations, the storage controller identifies a backup item that includes a set of manifests listed in the manifest list 445 (e.g., by parsing the identifiers of the manifests to extract the identifiers of the backup item that includes the manifests). The storage controller then compares the identified backup item to the item catalog 450 and determines that the manifest "K-7" is included in the backup item "K" that has a "retired" status in the item catalog 450 (e.g., has been marked for deletion). The storage controller then loads a copy of the manifest "K-7" from the persistent storage 140 to the memory 115 (e.g., via a read I / O operation) and reads the loaded manifest "K-7" to identify one or more ranges of data units to decrement. Further, the storage controller decrements the reference counts of the identified ranges of data units (in the data unit metadata 442 of the container index 160).

[0083] In another example, referring to Figure 4A and Figure 4D While the container index 160 remains loaded into the memory 115 for matching operations, the storage controller reads the reverse reference list 448 (in the container index 160) to identify a set of manifests indexed by the container index 160. The storage controller identifies a backup item that includes a set of manifests listed in the reverse reference list 448. The storage controller compares the identified backup item to the item catalog 450 and determines that the manifest "K-7" is included in the backup item "K" that has a "retired" status in the item catalog 450. The storage controller reads the entry for the manifest "K-7" in the reverse reference list 448 to identify a range of data units Rl. The storage controller identifies the reference counts (in the data unit metadata 442 of the container index 160) corresponding to the data units included in the range of data units Rl and then decrements the identified reference counts of the data units in the range of data units Rl.

[0084] Figure 7 - Example procedure for inlining in-house processing

[0085] Figure 7 An example process 700 for inline housekeeping is shown in accordance with some embodiments. In some examples, the process 700 can be performed using the storage controller 110 (as shown in FIG. 1). The process 700 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by a processor(s) ). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor memory device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. Figure 1

[0086] ​Block 710 can include receiving, by a storage controller of a deduplicated storage system, a stream of data units to be stored in a persistent storage of the deduplicated storage system. Block 720 can include loading, by the storage controller, a first container index into memory to index one or more data units included in the stream.

[0087] Block 730 can include updating, by the storage controller, the first container index loaded into memory to indicate a storage location of a first data unit included in the stream. Block 740 can include reading, by the storage controller, a manifest list in the first container index loaded into memory to identify a first manifest indexed by the first container index.

[0088] Block 750 can include determining, by the storage controller, whether the identified first manifest is included in a first backup item that has been marked for deletion. Block 760 can include decrementing, by the storage controller, a reference count of a second data unit in the first container index in response to determining that the identified first manifest is included in the first backup item that has been marked for deletion while the first container index remains loaded into memory to index one or more data units included in the stream, wherein the second data unit is identified in the first manifest.

[0089] Figure 8 - example machine-readable medium

[0090] Figure 8 A machine-readable medium 800 having instructions 810-850 stored therein is shown in accordance with some embodiments. The instructions 810-850 can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 500 can be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium.

[0091] The instructions 810 can be executed to receive a stream of data units to be stored in a persistent storage of a deduplicated storage system. The instructions 820 can be executed to load a first container index into memory to index one or more data units included in the stream. The instructions 830 can be executed to update the first container index loaded into memory to indicate a storage location of a first data unit included in the stream.

[0092] Instruction 840 can be executed to read a manifest list in a first container index loaded into memory to identify a first manifest indexed by the first container index. Instruction 850 can be executed to decrement a reference count of a second data unit in the first container index in response to determining that the identified first manifest is included in a first backup item that has been marked for deletion while the first container index remains loaded into memory to index one or more data units included in the stream, where the second data unit is identified in the first manifest.

[0093] According to some embodiments of the present disclosure, a controller of a deduplication storage system can perform inline housekeeping during a data ingest process. In some examples, when a container index is loaded into memory during a data ingest process, the controller can access a data structure in the container index to identify manifests indexed by the container index. The controller can identify each backup item that includes at least one of the manifests indexed by the container index and can determine whether the backup item has been marked for deletion. If so, the controller can perform an inline housekeeping process to decrement a reference count in the container index representing a data unit included in the marked backup item. Accordingly, because the inline housekeeping of the marked backup item is performed while the container index is already loaded into memory, a separate I / O operation to load the container index into memory to perform housekeeping of the marked backup item is not needed. In this way, some embodiments can reduce the amount of I / O bandwidth used for housekeeping jobs, which can improve the performance of the storage system.

[0094] Note that although Figures 1-8 Various examples are illustrated, but embodiments are not limited in this regard. For example, reference is made to Figure 1 It is contemplated that the storage system 100 can include additional devices and / or components, fewer components, different components, different arrangements, etc. In another example, it is contemplated that the functionality of the storage controller 110 described above can be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0095] Data and instructions are stored in respective storage devices, which are implemented as one or more computer- or machine-readable or -accessible storage media. Storage media include different forms of memory including semiconductor memory devices, such as dynamic or static random access memories (DRAMs or SRAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs) and flash memories; magnetic disks such as fixed, floppy and removable disks; other magnetic media including tape; optical media such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices.

[0096] Note that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system with potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or article of manufacture). An article or article of manufacture may refer to any single or multiple manufactured components. The one or more storage media may be located in a machine that executes the machine-readable instructions, or at a remote site from which the machine-readable instructions can be downloaded for execution via a network.

[0097] In the foregoing description, numerous details have been set forth to facilitate understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A computing device, comprising: processor; Memory; as well as A machine-readable storage device storing instructions that can be executed by the processor to perform the following operations: Receive a stream of data units to be stored in the persistent storage device of a deduplication storage system; A first container index is loaded into the memory to perform data ingestion on a set of received data units included in the stream, wherein the first container index is kept loaded into the memory during the data ingestion to perform deduplication on the set of received data units; During the data ingestion of the set of received data units, the first container index loaded into the memory is updated to indicate the storage location of the first data unit included in the set of received data units; During the data ingestion of the set of received data units, a list of lists loaded into the memory in the first container index is read to identify a first list indexed by the first container index; During the data ingestion of the set of received data units, it is determined whether the identified first list is included in the first backup item that has been marked for deletion; and In response to determining that the identified first list is included in the first backup item that has been marked for deletion, during the data ingestion of the set of received data units, the reference count of the second data unit identified in the first list is decremented in the first container index loaded into the memory.

2. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: During the data ingestion of the set of received data units, the following operations are performed in response to determining that the identified first list is included in the first backup item that has been marked for deletion: Load the first list into the memory; Read the loaded first list to identify one or more data unit ranges to be decremented; and Decrement the reference count of one or more identified data unit ranges in the first container index.

3. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: During the data ingestion of the set of received data units, the following operations are performed in response to determining that the identified first list is included in the first backup item that has been marked for deletion: Read entries from the list of the container index to identify one or more data unit ranges to be decremented; and Decrement the reference count of one or more identified data unit ranges in the first container index.

4. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: In response to determining that a housekeeping job has been scheduled in the job queue, the scheduled housekeeping job is executed in the deduplication storage system.

5. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: Access the list of lists in the first container index to identify a set of lists indexed by the first container index, wherein the first container index includes at least a first part and a second part, wherein the first part stores data unit metadata, wherein the second part stores the list of lists, wherein the list of lists includes one or more list entries, and wherein each list entry stores a list identifier for a different list for the deduplication storage system; Identify a set of backup items that include the identified list; as well as The current status of each of the set of backup projects is determined based on the project directory.

6. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations after the reference count of the second data unit in the first container index is decremented: Compress the first container index; and The first container index is saved to the persistent storage device.

7. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations after the reference count of the second data unit in the first container index is decremented: Update the project metadata of the first backup project to remove it from the list of the first container index; and In response to determining that the data ingestion session of the data unit's stream has been completed, the updated item metadata is saved to persistent storage.

8. A method for Deduplication storage system Methods, including: The storage controller of the deduplication storage system receives a stream of data units to be stored in the persistent storage device of the deduplication storage system; The storage controller loads a first container index into the memory to perform data ingestion on a set of received data units included in the stream, wherein the first container index remains loaded into the memory during the data ingestion to perform deduplication on the set of received data units; During the data ingestion of the set of received data units, the storage controller updates the first container index loaded into the memory to indicate the storage location of the first data unit included in the set of received data units; During the data ingestion of the set of received data units, the storage controller reads the list of lists loaded into the memory from the first container index to identify the first list indexed by the first container index; During the data ingestion of the set of received data units, the storage controller determines whether the identified first list is included in the first backup item that has been marked for deletion; During the data ingestion of the set of received data units, in response to determining that the identified first list is included in the first backup item that has been marked for deletion, the reference count of the second data unit in the first container index loaded into the memory is decremented by the storage controller, wherein the second data unit is identified in the first list.

9. The method of claim 8, comprising: During the data ingestion of the set of received data units, the following operations are performed in response to determining that the identified first list is included in the first backup item that has been marked for deletion: Load the first list into the memory; Read the loaded first list to identify one or more data unit ranges to be decremented; and Decrement the reference count of one or more identified data unit ranges in the first container index.

10. The method of claim 8, comprising: During the data ingestion of the set of received data units, the following operations are performed in response to determining that the identified first list is included in the first backup item that has been marked for deletion: Read entries from the list of the container index to identify one or more data unit ranges to be decremented; and Decrement the reference count of one or more identified data unit ranges in the first container index.

11. The method of claim 8, comprising: In response to determining that a housekeeping job has been scheduled in the job queue, the scheduled housekeeping job is executed in the deduplication storage system.

12. The method of claim 8, comprising: Access the list of lists in the first container index to identify a set of lists indexed by the first container index, wherein the first container index includes at least a first part and a second part, wherein the first part stores data unit metadata, wherein the second part stores the list of lists, wherein the list of lists includes one or more list entries, and wherein each list entry stores a list identifier for a different list for the deduplication storage system; Identify a set of backup items that include the identified list; as well as The current status of each of the set of backup projects is determined based on the project directory.

13. The method of claim 8, further comprising performing the following operation after decrementing the reference count of the second data unit in the first container index: Compress the first container index; and The first container index is saved to the persistent storage device.

14. The method of claim 8, further comprising performing the following operation after decrementing the reference count of the second data unit in the first container index: Update the project metadata of the first backup project to remove the list of the first container index; Determine whether the data ingestion session of the data unit's stream has been completed; and In response to determining that the data ingestion session of the data unit's stream has been completed, the updated item metadata is saved to persistent storage.

15. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to perform the following operations: Receive a stream of data units to be stored in the persistent storage device of a deduplication storage system; The first container index is loaded into the memory to perform data ingestion on a set of received data units included in the stream, wherein, The first container index is kept loaded into the memory during the data ingestion to perform deduplication on the set of received data units; During the data ingestion of the set of received data units, the index of the first container loaded into the memory is updated to indicate the storage location of the first data unit included in the stream; During the data ingestion of the set of received data units, a list of lists loaded into the memory in the first container index is read to identify a first list indexed by the first container index; During the data ingestion of the set of received data units, it is determined whether the identified first list is included in the first backup item that has been marked for deletion; as well as In response to determining that the identified first list is included in the first backup item that has been marked for deletion, during the data ingestion of the set of received data units, the reference count of the second data unit identified in the first list is decremented in the first container index loaded into the memory.

16. The non-transitory machine-readable medium of claim 15, comprising instructions that, upon execution, cause the processor to perform the following operations: During the data ingestion of the set of received data units, the following operations are performed in response to determining that the identified first list is included in the first backup item that has been marked for deletion: Load the first list into the memory; Read the loaded first list to identify one or more data unit ranges to be decremented; and Decrement the reference count of one or more identified data unit ranges in the first container index.

17. The non-transitory machine-readable medium of claim 15, comprising instructions that, upon execution, cause the processor to perform the following operations: During the data ingestion of the set of received data units, the following operations are performed in response to determining that the identified first list is included in the first backup item that has been marked for deletion: Read entries from the list of the container index to identify one or more data unit ranges to be decremented; and Decrement the reference count of one or more identified data unit ranges in the first container index.

18. The non-transitory machine-readable medium of claim 15, comprising instructions that, upon execution, cause the processor to perform the following operations: In response to determining that a housekeeping job has been scheduled in the job queue, the scheduled housekeeping job is executed in the deduplication storage system.

19. The non-transitory machine-readable medium of claim 15, comprising instructions that, upon execution, cause the processor to perform the following operations: Access the list of lists in the first container index to identify a set of lists indexed by the first container index, wherein the first container index includes at least a first part and a second part, wherein the first part stores data unit metadata, wherein the second part stores the list of lists, wherein the list of lists includes one or more list entries, and wherein each list entry stores a list identifier for a different list for the deduplication storage system; Identify a set of backup items that include the identified list; as well as The current status of each of the set of backup projects is determined based on the project directory.

20. The non-transitory machine-readable medium of claim 15, comprising instructions that, at execution time, cause the processor to perform the following operations after decrementing the reference count of the second data unit in the first container index: Compress the first container index; Save the first container index to the persistent storage device; Update the project metadata of the first backup project to remove it from the list of the first container index; and In response to determining that the data ingestion session of the data unit's stream has been completed, the updated item metadata is saved to persistent storage.

Images