REFERENCE DATA STRUCTURE FOR A DEDUPLICATION STORAGE SYSTEM
Patent Information
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- HEWLETT PACKARD ENTERPRISE DEV LP
- Filing Date
- 2023-10-04
- Publication Date
- 2026-07-30
Smart Images
Figure 00000000_0000_ABST
Abstract
Description
background Data reduction techniques can be applied to reduce the amount of data stored in a storage system. One example of a data reduction technique is data deduplication. Data deduplication identifies duplicate data units and attempts to reduce or eliminate the number of duplicate data units stored in the storage system. Short description of the drawings Some embodiments are described with reference to the following figures. Fig. 1 is a schematic diagram of an example memory system, according to some embodiments. Fig. 2 shows example data structures, according to some implementations. Figs. 3A-3B are illustrations of example processes, according to some implementations. Figs. 4A-4G are illustrations of example operations, according to some implementations. Figs. 5A-5E are illustrations of example operations, according to some implementations. Fig. 6 is an illustration of an example process, according to some implementations. Fig. 7 is a schematic diagram of an example computing device, according to some implementations. Fig. 8 is an illustration of an example process, according to some implementations.9 is a diagram of a machine-readable medium storing instructions in accordance with some implementations. In the drawings, identical reference numbers indicate similar, but not necessarily identical, elements. The illustrations are not necessarily to scale, and the size of some parts may be exaggerated to better illustrate the example shown. Furthermore, the drawings contain examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments shown in the drawings. Detailed description In this disclosure, the use of the term "a," "an," or "the" includes the plural forms unless the context clearly indicates otherwise. Likewise, the term "comprises," "including," "comprises," "includes," "has," or "have," when used in this disclosure, specifies the presence of the specified elements but does not preclude the presence or addition of other elements. In some examples, a storage system may back up a collection of data (referred to herein as a "data stream" or "data flow") in deduplicated form, thereby reducing the storage space required to store the data flow. The storage system may create a "backup item" to represent a data stream in deduplicated form. The storage system may perform a deduplication process that breaks a data stream into discrete data units (or "chunks") and determines "fingerprints" (see below) for those incoming data units. Further, the storage system may compare the fingerprints of incoming data units with the fingerprints of stored data units, thereby determining which incoming data units are duplicates of previously stored data units (e.g., if the comparison results in matching fingerprints).In the case of data units that are duplicates, the storage system can store references to previously stored data units instead of storing the duplicate incoming data units. As used here, the term "fingerprint" refers to a value derived by applying a function to the contents of the data unit (where the "contents" may include all or a subset of the contents of the data unit). An example of a function that may be applied is a hash function, which generates a hash value based on the contents of an incoming data unit. Examples of hash functions include cryptographic hash functions such as the Secure Hash Algorithm 2 (SHA-2) hash functions, e.g., SHA-224, SHA-256, SHA-384, etc. In other examples, other types of hash functions or other types of fingerprint functions may also be used. A "storage system" may include a storage device or an array of storage devices. A storage system may also include one or more storage controllers that manage access to the storage device(s). A "unit of data" may refer to any piece of data that can be separately identified in the storage system. In some cases, a unit of data may refer to a chunk, a collection of chunks, or any other piece of data. In some examples, a storage system may store units of data in persistent storage. Persistent storage may be achieved using one or more persistent (e.g., non-volatile) storage devices, such as disk-based storage devices (e.g., hard disk drives (HDDs)), solid-state devices (SSDs), such as flash memory devices, or the like, or a combination thereof. A "control device" may refer to a hardware processing circuit, which may include any one or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuit. Alternatively, a "control device" may refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry. In some examples, a deduplication storage system may use metadata for processing incoming data streams (e.g., backup items). Such metadata may include, for example, data recipes (also referred to herein as "manifests") that specify the order in which specific data units are received for each backup item. Furthermore, such metadata may include item metadata that represents each received backup item (e.g., a data stream) in deduplicated form. The item metadata may contain identifiers for a set of manifests and specify the sequential order of the manifest set. The processing of each backup item may be referred to herein as a "backup process."Then, in response to a read request, the deduplication system can use the item's metadata and the set of manifests to determine the received order of the data units, thus restoring the original data stream of the backup item. Accordingly, the set of manifests can be a representation of the original backup item. The manifests can contain a sequence of records, where each record represents a specific set of data units. The manifest records can contain one or more fields that identify container indexes that index the data units (e.g., contain storage information for them). For example, a container index can contain one or more fields that specify location information (e.g., container, offsets, etc.) for the stored data units, compression and / or encryption characteristics of the stored data units, etc.In addition, the container index can contain reference counts that indicate the number of manifests that point to each data unit. In some examples, upon receipt of a piece of data (e.g., in a data stream), it may be compared against one or more container indexes to determine if an identical chunk is already stored in a container of the deduplication storage system. For example, the deduplication storage system may compare the fingerprint of the received piece of data with the fingerprints in one or more container indexes. If no matching fingerprints are found in the scanned container indexes, the received piece of data may be added to a container, and an entry for the received piece of data may be added to a container index corresponding to that container. However, if a matching fingerprint is found in a scanned container index, it may be determined that a piece of data identical to the received piece of data is already stored in a container.In response to this determination, the reference count of the corresponding entry is incremented, and the received data unit is not stored in a container (because it already exists in one of the containers), thus avoiding storing a duplicate data unit in the deduplication storage system. The term "matching" as used here can refer to a process of comparing fingerprints of a collection of multiple data units (e.g., from a specific backup stream) with fingerprints stored in a container index. In some examples, the deduplication storage system may perform housekeeping tasks or processes to maintain the accuracy of the contained metadata. For example, if a data unit is deleted from a particular manifest (e.g., due to a change in the data stream or the element represented by the manifest), a housekeeping job may include decrementing the reference count for that data unit by one (i.e., indicating that the data unit is referenced by one less manifest). In some examples, a housekeeping job to delete a backup item may include identifying the manifests contained in the backup item, accessing those manifests sequentially and loading them into memory, and reading the manifests to determine the container identifiers and address information for the data units in each manifest.The housekeeping job may also include accessing and loading the container indexes (e.g., using the container identifiers) into memory, decrementing the reference counts for the data units (e.g., using the address information for the data units), and then storing the container indexes in persistent storage. Accordingly, performing such housekeeping jobs may require a relatively large number of input / output (I / O) operations to load multiple metadata items (e.g., manifests and container indexes) from persistent storage into memory and to write at least some of the multiple metadata items from memory to persistent storage. Accordingly, performing such housekeeping tasks may reduce the performance of the deduplication storage system. In accordance with some implementations of the present disclosure, a controller of a deduplication storage system may create a work summary for performing a cleanup job (e.g., to delete a backup item that is no longer needed). An entry in the work summary (also referred to as a "work entry") may identify a container index and a manifest. The controller may select a work entry for processing and then load the container index identified in the work entry from persistent storage into storage. The controller may access or read a back-reference data structure contained in the container index. Each entry of the back-reference data structure (also referred to as a "back-reference entry") may specify a different manifest and a data unit range.The data unit range may refer to a set of data units contained in the specified manifest and indexed by the container index. The controller may select the backreference entry that specifies the same manifest as specified in the work entry and extract the data unit range specified in the selected backreference entry. The controller may then decrement the reference counts in the container index corresponding to the extracted data unit range. In this way, the work entry may be processed without loading the manifest into memory. Accordingly, some implementations may reduce the amount of I / O bandwidth used for housekeeping jobs, thereby improving storage system performance. Various aspects of the disclosed housekeeping process are described below with reference to Figures 1-9. FIG. 1 - Example of a storage system Fig. 1 shows an example of a storage system 100 that includes a storage controller 110, a memory 115, and persistent storage 140, in accordance with some implementations. The persistent storage 140 may include one or more non-transitory storage media such as hard disk drives (HDDs), solid-state drives (SSDs), optical disks, etc., or a combination thereof. The memory 115 may be implemented in semiconductor memory such as random access memory (RAM). In some examples, the memory controller 110 may be implemented by hardware (e.g., electronic circuits) or a combination of hardware and programming (e.g., at least one processor and instructions executable by the at least one processor stored on at least one machine-readable storage medium). As shown in Fig. 1, the memory 115 and the persistent storage 140 may store various data structures, including at least one work map 120, item metadata 130, manifests 150, container indexes 160, and data containers 170. In some examples, copies of the manifests 150, container indexes 160, and data containers 170 may be transferred between the memory 115 and the persistent storage 140 (e.g., via read and write input / output (I / O) operations). In some implementations, storage system 100 may perform deduplication of the stored data. For example, storage controller 110 may receive an incoming data stream 105 (also referred to herein as a "backup element") containing multiple data units and store at least one copy of each data unit in a data container 170 (e.g., by appending the data units to the end of data container 170). In some examples, each data container 170 may be divided into units 175, where each unit 175 contains multiple stored data units. In one or more implementations, storage controller 110 may generate a fingerprint for each received data unit. The fingerprint may include, for example, a full or partial hash value based on the data unit. To determine whether an incoming data unit is a duplicate of a stored data unit, storage controller 110 may compare the fingerprint generated for the incoming data unit with the fingerprints in at least one bin index. If a match is found, storage controller 110 may determine that a duplicate of the incoming data unit is already stored in storage system 100. Storage controller 110 may then store references to the previous data unit instead of storing the duplicate incoming data unit. In some implementations, the storage controller 110 may generate item metadata 130 to represent each received backup item (e.g., a data stream 105) in deduplicated form. Each item metadata 130 may contain identifiers for a set of manifests 150 and indicate the sequential order of the set of manifests 150. The manifests 150 record the order in which the data units were received. Additionally, the manifests 150 may contain a pointer or other information indicating the container index 160 that indexes each data unit. In some implementations, the container index 160 may indicate the location where the data unit is stored. For example, the container index 160 may contain information indicating that the data unit is stored at a particular offset in an entity and that the entity is stored at a particular offset in a data container 170.In addition, the container index 160 can contain reference numbers that indicate the number of manifests 150 that point to each data unit. In some implementations, storage controller 110 may receive a read request to access the stored data and then access item metadata 130 and manifests 150 to determine the order of the data units that comprised the original data. Storage controller 110 may then use the pointer data contained in a manifest 150 to identify the container indices 160 that index the data units. Further, storage controller 110 may use the information contained in the identified container indices 160 (and the information contained in the manifest 150) to determine the storage locations of the data units (e.g., data container 170, entity 175, offsets, etc.) and may then read the data units from the determined storage locations. In some implementations, storage controller 110 may use work summary 120 to perform housekeeping tasks of storage system 100. For example, storage controller 110 may receive a request to delete a particular backup item and then schedule a housekeeping job to run at a later time. After starting the housekeeping job, storage controller 110 may load the item metadata 130 for the backup item from persistent storage 140 into storage 115. In some implementations, storage controller 110 may use the item metadata 130 to identify the manifests 150 representing the backup item and to identify the container indexes 160 referenced by the identified manifests 150.The storage controller 110 may populate the work map 120 with a set of work entries corresponding to the container indexes 160 identified in the item metadata 130. Each work entry may identify the corresponding container index 160 and may also identify a set of manifests 150, each of which points to the corresponding container index 160. Some example implementations of the work map 120 and the item metadata 130 are described below with reference to FIGS. 4A-5E. In some implementations, the storage controller 110 may select a work item from the work summary 120 for processing and then load the container index 160 identified in the work item from the persistent storage 140 into the storage 115. The container index 160 may include a data structure for storing a set of backreference entries, where each backreference entry specifies a different manifest 150 and a data unit range (e.g., information for identifying a set of data units included in the manifest 150 and indexed by the container index 160). The storage controller 110 may select the backreference entry that specifies the same manifest 150 identified in the work item and may extract the data unit range specified in the selected backreference entry.The data unit range (extracted from the back-reference entry) can be used to deterministically identify one or more data units with reference counts to be decremented for the housekeeping job. The storage controller 110 can then decrement the reference counts in the container index 160 corresponding to the one or more data units identified by the extracted data unit range. In this way, the use of the back-reference entries in the container index 160 can enable the corresponding reference counts for the housekeeping job to be decremented without requiring the manifests 150 to be loaded into memory to identify the one or more data units associated with those reference counts.Accordingly, some implementations may reduce the amount of I / O bandwidth used for housekeeping jobs, thereby improving the performance of the deduplication storage system 100. In some implementations, the storage controller 110 may use the item metadata 130 to obtain, for each combination of a manifest 150 and a container index 160 (also referred to as a "manifest-CI combination") identified in the item metadata 130, a Boolean flag (also referred to as a "backreference flag" or "BR flag") that indicates whether that particular manifest-CI combination should be processed using the backreference entries contained in the container index 160. The storage controller 110 may populate the work entries of the work summary 120 based on the backreference flags associated with each manifest. For example, if the backreference flag is set to a "false" value (i.e.indicates that the manifest-CI combination should not be processed using a backreference entry), the memory controller 110 loads the manifest 150 into memory 115 to determine the data unit ranges corresponding to the reference counts to be decremented in the container index 160, and then appends the determined data unit ranges and an identifier of the manifest 150 to the working entry (e.g., in a data field). Otherwise, if the backreference flag is set to a "true" value (i.e., indicating that the manifest-CI combination should be processed using a backreference entry), the memory controller 110 appends the identifier of the manifest 150 to the working entry, but does not load the manifest 150 into memory 115 and does not append the data unit ranges to the working entry.Example processes for performing housekeeping using backlookup data structures are described below with reference to Figures 3A-3B. FIG. 2 - Example data structures Fig. 2 shows an example of data structures 200 used in deduplication, according to some implementations. As shown, the data structures 200 may include item metadata 202, a manifest 203, a container index 220, and a data container 250. In some examples, the item metadata 202, the manifest 203, the container index 220, and the data container 250 may generally correspond to the example implementations of item metadata 130, a manifest 150, a container index 160, and a data container 170 (shown in Fig. 1), respectively. In some examples, the data structures 200 may be created and / or managed by the storage controller 110 (shown in Fig. 1). In some implementations, the metadata 202 may include multiple manifest identifiers 205. Each manifest identifier 205 may identify a different manifest 203. In some implementations, the manifest identifiers 205 may be arranged in a stream order (i.e., based on the order of receipt of the data units represented by the identified manifests 203). Furthermore, the item metadata 202 may include a container list 204 associated with each manifest identifier 205. In some implementations, the container list 204 may include identifiers for a set of container indices 220 that index the data units contained in the associated manifest 203 (i.e., the manifest 203 identified by the associated manifest identifier 205). Furthermore, in some embodiments, the container list 204 may include a backreference flag associated with each container index identifier. The backreference flag may indicate whether the manifest CI combination (corresponding to the associated container index identifier and the manifest identifier 205) should be processed using the backreference list 222 (see below) contained in the container index 220. An example implementation of the container list 204 is described below with reference to Figures 4A-4G. Although only one of them is shown in Fig. 2 for simplicity, the data structures 200 may contain a plurality of instances of item metadata 202, each containing or referencing one or more manifests 203. In such examples, the data structures 200 may contain a plurality of manifests 203. The manifests 203 may refer to a plurality of container indexes 220, each corresponding to one of a plurality of data containers 250. Each container index 220 may contain one or more records 230 and one or more entity records 240. As shown in Figure 2, in some examples, each manifest 203 may include one or more manifest records 210. Each manifest record 210 may include various fields, such as offset, length, container index, and unit address. In some embodiments, each container index 220 may include any number of data unit records 230 and entity records 240. Each record 230 may include various fields, such as a fingerprint (e.g., a hash of the data unit), a unit address, an entity identifier, a unit offset (i.e., an offset of the data unit within the entity), a reference count, and a unit length. In some examples, the reference count may indicate the number of manifest records 210 that reference the data unit record 230. Additionally, each entity record 240 may include various fields, such as: B. an entity identifier, an entity offset (iean offset of the entity within the container), a stored length (i.e., a length of the data unit within the entity), a decompressed length, a checksum value, and compression / encryption information (e.g., type of compression, type of encryption, etc.). In some embodiments, each container 250 may contain any number of units 260, and each unit 260 may contain any number of stored data units. In some implementations, the unit address (contained in the manifest record 210 and the data unit record 230) may be an identifier that deterministically identifies a particular data unit within a particular container index 220. In some examples, the unit address may be a numeric value (referred to as an "arrival number") that indicates the sequential order of arrival (also referred to as the "ingest order") of data units that are indexed in a particular container index 220 (e.g., when receiving and deduplicating an incoming data stream). For example, the first data unit to be indexed in a container index 220 (e.g.,The first data unit may be assigned the arrival number "1" (e.g., by creating a new data unit record 230 for the first data unit), the second data unit may be assigned the arrival number "2," the third data unit may be assigned the arrival number "3," and so on. However, other implementations are also possible. In some implementations, a manifest record 210 may use a run-length reference format to represent a continuous range of data units (e.g., a portion of a data stream) indexed within a single container index 220. The run-length reference may be recorded in the unit address field and the length field of the manifest record 210. For example, the unit address field may specify the arrival number of a first data unit in the represented data unit range, and the length field may specify a number N (where "N" is an integer) of data units in the data unit range that follow the data unit specified by the arrival number in the unit address field. The data units in a data unit range may have consecutive arrival numbers (e.g., because they follow one another in a read-in data stream).Thus, a data unit range may be represented by an arrival number of a first data unit in the data unit range (e.g., specified in the unit address field of a manifest record 210) and a number N of further data units in the data unit range (e.g., specified in the length field of the manifest record 210). The further data units in the data unit range after the first data unit may be deterministically derived by calculating the N arrival numbers that consecutively follow the specified arrival number of the first data unit, where these N arrival numbers identify the further data units in the data unit range.In such examples, the manifest record 210 may include an arrival number "X" in the unit address field and a number N in the length field to specify a data unit range that includes the data unit specified by the arrival number X and the data units specified by the arrival numbers X+i for i=0 to i=N, inclusive (where "i" is an integer). In this way, the manifest record 210 may be used to identify all data units in the data unit range. In one or more implementations, data structures 200 may be used to retrieve stored deduplicated data. For example, a read request may specify an offset and length of data in a particular file. These request parameters may be matched to the offset and length fields of a particular manifest record 210. The container index and unit address of the particular manifest record 210 may then be matched to a particular data unit record 230 contained in a container index 220. Further, the entity identifier of the particular record 230 may be matched to the entity identifier of a particular entity record 240. In addition, one or more other fields of the particular entity record 240 (e.g., the entity offset, stored length, checksum, etc.) may be matched.) to identify the container 250 and the entity 260, and the data unit can then be read from the identified container 250 and the entity 260. In some embodiments, each container index 220 may include a backreference list 222. The backreference list 222 may be a data structure for storing a set of backreference entries, where each backreference entry specifies a different manifest 203 and at least one data unit range (e.g., a set of one or more data units included in the manifest 203 and indexed by the container index 220). In some embodiments, a backreference entry (in backreference list 222) may specify a data unit range as a run-length reference that identifies a first data unit and the length of the data unit range in container index 220. For example, the run-length reference may identify the first data unit in the data unit range using a first arrival number "X" assigned to the first data unit in container index 220. Further, the run-length reference may identify a length value N that indicates the number of arrival numbers consecutively following the first arrival number "X" that are assigned in container index 220 to the remaining data units in the data unit range.In such examples, the run-length reference contained in the backreference entry can be used to identify the first data unit in the range (with the first arrival number "X") and then to identify the remaining data units in the range (with the N arrival numbers that sequentially follow the first arrival number "X"). In this way, the run-length reference contained in the backreference specification can be used to deterministically identify all data units in the data unit range. In other implementations, the backreference entry may specify a data unit range (including one or more data units) using other identifiers or representations. For example, the backreference entry may specify a data unit range as a fingerprint of a single data unit, as a set of fingerprints for multiple data units, as a set of arrival numbers for multiple data units, as an offset (e.g., one byte from an end), as an offset and length, or the like. In some implementations, when initiating a housekeeping job to delete a backup item, a storage controller (e.g., storage controller 110 shown in Figure 1) may load the item metadata 202 for the backup item into storage. The storage controller may use the item metadata 202 to identify a particular manifest 203 representing the backup item and to identify at least one container index 220 referenced by the identified manifest 203. The storage controller may load the identified container index 220 into storage and then access the backreference list 222 contained in the identified container index 220. Further, the storage controller may identify the backreference entry (in the backreference list 222) specifying the particular manifest 203 and the data unit scope information (e.g.,a run-length reference) contained in the identified backreference entry. The storage controller may then decrement the reference numbers in the container index 220 that correspond to the data units identified by the data unit range information. In this way, use of the backreference list 222 in the container index 220 may enable the decrementing of the corresponding reference numbers for the housekeeping job without requiring the respective manifest 203 to be loaded into memory to identify the one or more data units associated with those reference numbers. Accordingly, some implementations may reduce the amount of I / O bandwidth used for housekeeping jobs, thereby improving the performance of the deduplication storage system. In some embodiments, each time the container index 220 is created or updated to include information about a particular manifest 203, the backreference list 222 in that container index 220 is updated to identify that manifest 203 and at least one data unit range that identifies the data units of the manifest 203 that are indexed by the container index 220. When the container index 220 is no longer associated with a manifest 203, the backreference entry for that manifest 203 may be removed from the backreference list 222. An example implementation of the backreference list 222 is described below with reference to FIGS. 5A-5E. FIGS. 3A and 4A-4G - Example processes for creating a job card Fig. 3A shows an example process 300 for creating a housekeeping work plan, in accordance with some implementations. For illustrative purposes, the details of process 300 are described below with reference to Figs. 4A-4G, which show example operations in accordance with some implementations. However, other implementations are possible. In some examples, process 300 may be performed using memory controller 110 (shown in Fig. 1). Process 300 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device.The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some implementations, process 300 may be executed by a single processing thread. In other implementations, process 300 may be executed in parallel by multiple processing threads (e.g., concurrently using the work map and executing multiple housekeeping jobs). Block 310 may include scheduling housekeeping jobs in a job queue. Block 312 may include initiating a housekeeping job to delete a backup item stored in a deduplication storage system. Job queue 410 is a data structure in which requests to delete backup items can be queued (see Fig. 4A). The storage controller (e.g., storage controller 110 in Fig. 1) receives multiple delete requests to delete backup items AC stored in a deduplication storage system. In response to the received requests, the storage controller schedules multiple housekeeping jobs in a job queue 410 to delete the backup items AC. The storage controller selects a job from job queue 410 and initiates the selected job to delete backup item A. Block 314 may include loading item metadata into storage. For example, the storage controller 110 (see FIGS. 1 and 4A-4B) loads the metadata 420 for Item A (i.e., the metadata for Backup Item A) from the persistent storage 140 into the storage 115. The Item A metadata 420 includes a plurality of manifest identifiers 422 and a plurality of container lists 425, with each container list 425 associated with a different manifest identifier 422. Each container list 425 contains a set of CI entries. Each CI entry contains a CI identifier for each container index 160, which indexes data units included in the manifest 150 identified by the associated manifest identifier 422.In other words, each CI entry represents a different manifest-CI combination, namely a combination of the manifest 150 (identified by the manifest identifier 422) and the container index 160 (identified by the CI identifier in the CI entry). Furthermore, each CI entry contains a backreference flag ("BR flag") that indicates whether the manifest-CI combination (represented by the CI entry) should be processed using the backreference data structure contained in the container index 160. Block 316 may include selecting a container index (CI) entry from the container lists contained in the location metadata. Block 318 may include determining the container index, manifest, and backreference flag (BR) for the selected CI entry. For example, the storage controller accesses the item A metadata 420 to read the manifest identifier MA-1 422 and the associated container list 425 (see Figure 4B). Further, the storage controller reads a first CI entry (in the container list 425) that includes the container index identifier C-3. Furthermore, the first CI entry includes a backreference flag set to true ("T"), indicating that the combination of manifest MA-1 and container index C-3 should be processed using the backreference data structure contained in the container index C-3. At decision block 320, it may be determined whether the work map already contains a work entry associated with the identified container index (i.e., the container index identified in the CI entry selected in field 316). If so ("YES"), the process 300 may continue at decision block 324 (see below). Otherwise, if it is determined at decision block 320 that the work map does not contain a work entry associated with the identified container index ("NO"), the process 300 may continue at block 322, including creating a new work entry associated with the identified container index. For example, the storage controller (see Figure 4B) determines that the work map 430 does not contain a work entry for container index C-3 (identified in the first CI entry of the container list 425).In response to this determination, the storage controller creates a new work entry for C-3 (i.e., a new work entry indexed by the container index C-3) in the work summary 430. In another example, referring to Fig. 4D, the storage controller determines that the work summary 430 does not contain a work entry for the container index C-4 (identified in the second CI entry of the container list 425). In response to this determination, the storage controller creates a new work entry for C-4. At decision block 324, a determination may be made as to whether the backreference flag in the selected CI entry is set to a true value. If so ("YES"), the process 300 may continue to block 330, including adding the manifest identifier and backreference flag to the work entry for the identified container index. For example, in Fig. 4C, the warehouse manager determines that the CI entry for C-3 contains a backreference flag set to "true" ("T"). In response to this determination, the warehouse manager inserts the manifest identifier "MA-1" and the backreference flag "BR=T" into a first field ("Field1") of the work entry for C-3 (in the work summary 430). However, if it is determined at decision block 324 that the backreference flag in the selected CI entry is not set to a true value ("NO"), the process 300 may continue at block 326, including loading the identified manifest into memory and reading the identified manifest to determine the data unit range(s) for the identified container index. Block 328 may include adding the manifest identifier, the address information for the data unit range(s), and the backreference flag to the working entry for the identified container index. For example, in Fig. 4E, the storage controller determines that the CI entry for C-4 contains a backreference flag set to false ("F"). In response to this determination, the storage controller loads the manifest MA-1 440 from the persistent store 140 into the storage 115.The storage controller reads the manifest MA-1 440 to determine the data unit range R2 for the reference counts to be decremented in the container index C-4. The storage controller then adds the manifest identifier "MA-1," data identifying the data unit range R-2, and the backreference flag "BR=F" to a first field of the work entry for C-4. The data identifying the data unit range R-2 may include, for example, one or more run-length references, data unit fingerprints, data unit arrival numbers, or the like. At decision block 332, a determination may be made as to whether the job card is full (e.g., has reached a maximum number of work entries) or whether there are no more CI entries remaining in the item metadata container lists. If not ("NO"), the process 300 may return to block 316 (e.g., to select another CI entry from the item metadata container lists). Otherwise, if it is determined at decision block 332 that the job card is full or that no more CI entries remain in the container lists, the process 300 may continue at block 334, including processing the job card to complete at least a portion of the housekeeping job. An exemplary extension of block 334 (i.e., an exemplary process for processing the job card) is described below with reference to FIGS. 3B and 5A-5E. Example: After creating work items for container indexes C-3 and C-4, the storage controller determines that the work map 430 already contains a work item for container index C-2 (identified in the third CI entry of the container list 425) (see Fig. 4F). In response to this determination, the storage controller adds the manifest identifier "MA-1" and the backreference flag "BR=T" to a second field ("Field2") of the existing work item for C-2. In another example, referring to Fig. 4G, the storage controller determines that the work map 430 does not contain a work item for container index C-7 (identified in the fourth CI entry of the container list 425). In response to this finding, the memory controller creates a new working entry for C-7 and adds the manifest identifier “MA-1” and the backreference flag “BR=T” (iea true value) to a first field (“Field2”) of the new work entry for C-7. Referring again to Fig. 3A , after processing the job card (at block 334), the process 300 may proceed to decision block 336, including determining whether any additional CI entries remain in the item metadata container lists. If so ("YES"), the process 300 may return to block 316 (e.g., to select another CI entry from the item metadata container lists). Otherwise, if it is determined at decision block 336 that no additional CI entries remain in the container lists ("NO"), the process 300 may continue to block 338, including deleting the manifests identified in the item metadata. Block 340 may include deleting the item metadata. After block 340, the process 300 may return to block 312 (e.g., to initiate another housekeeping job to delete another backup item). For example, see Fig.1 and 4G: The storage controller 110 deletes each manifest 150 identified in the item A metadata 420 from the persistent storage 140 and the storage 115. In addition, the storage controller 110 deletes the item A metadata 420 from the persistent storage 140 and the storage 115. FIGS. 3B and 5A-5E - Example processes for processing a job card Fig. 3B shows an example process 334 for processing a janitor job card in accordance with some implementations. Process 334 may represent an example extension of block 334 (shown in Fig. 3A). Accordingly, in some implementations, process 334 may be performed after determining that a job card is full or no more CI entries exist in the item metadata container lists (at decision block 332 in Fig. 3A). For illustrative purposes, the details of method 334 are described below with reference to FIGS. 1 and 5A-5E. However, other implementations are possible. In some examples, process 334 may be performed using memory controller 110 (shown in FIG. 1). Process 334 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.In some implementations, process 334 may be executed by a single processing thread. In other implementations, process 334 may be executed by multiple processing threads in parallel. At block 350, an outer loop (defined by blocks 350-354) may be initiated to process each work item in a housekeeping work map. Block 352 may include loading the container index identified in the current work item into storage. For example, the storage controller (e.g., storage controller 110 in Fig. 1) traverses the work items of work map 420 (see Fig. 5A). The storage controller selects a first work item of work map 420 and reads an identifier for container index C-2. The storage controller then causes container index C-2 510A to be loaded from persistent storage 140 into storage 115. At block 354, an inner loop (defined by blocks 354-362) may be initiated to process each manifest identified in the current work item (i.e., the work item being processed in the current iteration of the outer loop). At decision block 356, a determination may be made as to whether the backreference flag associated with the current manifest is set to a true value. If so ("YES"), process 334 may continue at block 360, including performing a lookup operation for the identified manifest in a backreference list of the loaded container index to identify the data unit range(s). Block 362 may include decrementing the reference counts in the loaded container index for the data unit range(s) identified in the backreference list.After block 362, process 334 may return to block 354 to perform another iteration of the inner loop (e.g., to process another manifest in the work item). After all iterations of the inner loop are complete (e.g., all manifests in the current work item have been processed), process 334 may continue to block 364, including saving the container index to persistent storage. After block 364, process 334 may return to block 350 to perform another iteration of the outer loop (e.g., to process another work item in the work summary). For example, the storage controller reads the first field ("Field1") in the first working entry (for container index C-2), thereby obtaining the manifest identifier "MA-5" and the backreference flag "BR=T" (i.e., a true value). The storage controller compares the manifest identifier "MA-5" with the third backreference entry in the backreference list 530A of container index C-2 510A and reads the data region information in the third backreference entry to identify data unit region R1. The storage controller identifies the reference counts (in the data unit metadata 520A of container index C-2 510A) that correspond to the data units contained in data unit region R1. The storage controller then decrements the identified reference counts for the data units in data unit region R1. In some examples, the data region information contained in backreference list 530A may be specified as a run-length reference.In such examples, the memory controller reads the run-length reference (in the backreference entry) to identify the first data unit in the data unit range R1 (e.g., with a first arrival number "X") and then identifies the remaining data units in the data unit range R1 (e.g., with the N arrival numbers sequentially following the first arrival number "X"). In this way, the memory controller uses the run-length reference in the backreference entry to identify all data units in the data unit range R1 and then decrements the reference counts (in the data unit metadata 520A of the container index C-2 510A) corresponding to the identified data units in the data unit range R1.In this manner, the back-reference list 530A in the container index C-2 510A can be used to decrement the corresponding reference counts for the housekeeping order without having to load the manifest "MA-5" into memory to identify the data units associated with those reference counts. In another example, referring to Fig. 5C, the storage controller reads the second field ("Field2") in the first working entry (for container index C-2), thereby obtaining the manifest identifier "MA-1" and the backreference flag "BR=T." The storage controller compares the manifest identifier "MA-1" with the first backreference entry in the backreference list 530A, thereby obtaining the data unit range R3. The storage controller then decrements the reference counts (in the data unit metadata 520A of the container index C-2 510A) for the data units represented by the data unit range R3. After processing the first and second fields of the first work entry (for container index C-2), the storage controller stores the container index C-2 510A in persistent storage and then selects the second work entry (for container index C-3) of the work summary 420 to be processed. If it is determined at decision block 356 that the backreference flag associated with the current manifest is not set to a true value ("NO"), the process 334 may continue at block 358, including decrementing the reference counts in the identified container index for the data unit range(s) listed in the current work entry. After block 358, the process 334 may return to block 354 to perform another iteration of the inner loop (e.g., to process another manifest in the work entry). For example, in Fig. 5D, the work map 430 is shown to represent a state after the first work entry (for container index C-2) and the second work entry (for container index C-3) have been processed and completed. The memory controller selects the next work entry (i.e., the third entry) of the work map 420 and reads an identifier for the container index C-4.The memory controller then causes the container index C-4 510B to be loaded from the persistent storage 140 into the memory 115. As shown in Figure 5E, the memory controller reads the first field in the third working entry (for the container index C-4), thereby obtaining the manifest identifier "MA-1," the backreference flag "BR=F" (i.e., a false value), and the data unit range R2 (e.g., stored in field 328 of Figure 3A). The memory controller then decrements the reference counts (in the data unit metadata 520B of the container index C-4 510B) for the data units represented by the data unit range R2. After all iterations of the outer loop are complete (e.g., all work items in the work map have been processed), process 334 may terminate (see Fig. 3B). Referring to Fig. 3A, for example, the storage controller may continue process 300 at decision block 336, including determining whether any additional CI entries remain in the item metadata container lists and, if so, selecting another CI entry to process using work map 120. FIG. 6 - Example process for creating metadata Fig. 6 shows an example process 600 for generating metadata in accordance with some implementations. For illustrative purposes, the details of the process 600 are described below with reference to Figs. 1 and 4A-5E, which show examples of some implementations. However, other implementations are possible. In some examples, the process 600 may be performed using the memory controller 110 (shown in Fig. 1). The process 600 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device.The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. Block 610 may include receiving a backup item to be stored in persistent storage of a deduplication storage system. Block 620 may include generating fingerprints for the data units of the received backup item. For example, storage controller 110 receives a backup item (e.g., data stream 105) to be stored in deduplication storage system 100 and generates fingerprints for the data units in the received backup item. Block 630 may include comparing the generated fingerprints with fingerprints stored in container indexes of the deduplication storage system. Block 640 may include identifying a subset of the generated fingerprints that do not match the fingerprints stored in the container indexes. Block 650 may include storing the identified subset of generated fingerprints in a set of container indexes. For example, the storage controller 110 compares the generated fingerprints with the fingerprints contained in the container indexes 160 (see Fig. 1). If a match is found for a data unit, the storage controller 110 determines that a duplicate of the data unit is already stored in the storage system 100. In response to this determination, the storage controller 110 stores a reference to the previous data unit (e.g.,in a manifest 150) in deduplicated form. Otherwise, if no match is identified for a data unit, the storage controller 110 stores the data unit in a data container 170 and adds an entry for the data unit to a container index 160 corresponding to that data container 170. Block 660 may include creating a set of manifests to record the order of the data units of the received backup item. Block 670 may include storing a backreference list in each container index to record the manifests and data unit ranges that reference the container index. For example, storage controller 110 records the order in which the data units are received in one or more manifests 150 (see FIGS. 1 and 5A). Furthermore, storage controller 110 creates backreference list 445 in container index C-2 440. Backreference list 445 identifies each manifest 150 (e.g., MA-1) that currently references container index C-2 400.Furthermore, for each identified manifest 150, the backreference list 445 may identify at least one data unit range representing the set of data units included in the manifest 150 and indexed by the container index C-2 400. In some implementations, the storage controller 110 may continuously update the manifest list 445 during operation (e.g., to account for deletions or additions of manifests to the manifest list 445 during changes to the data stored in the storage system 100). Block 680 may include generating metadata for an object to record the identifiers of the set of manifests. Block 690 may include storing at least one container list in the object's metadata to record the set of container indices and associated backreference identifiers. After block 690, process 600 may be complete. For example, referring to FIGS. 1 and 4B, storage controller 110 generates item A metadata 420 to represent backup item A. Item A metadata 420 includes a plurality of manifest identifiers 422 and a plurality of container lists 425, each container list 425 associated with a different manifest identifier 422. Each container list 425 includes a number of CI entries. Each CI entry contains a CI identifier for each container index 160 that indexes data units contained in the manifest 150 identified by the associated manifest identifier 422.In addition, each CI entry contains a backreference flag that indicates whether the manifest-CI combination (represented by the CI entry) should be processed using the backreference data structure contained in container index 160. FIG. 7 - Example of a data processing system Fig. 7 shows a schematic diagram of an example computing device 700. In some examples, computing device 700 may generally correspond to part or all of storage system 100 (shown in Fig. 1). As shown, computing device 700 may include a hardware processor 702, a memory 704, and a machine-readable storage 705 having instructions 710-760. Machine-readable storage 705 may be a non-transitory medium. Instructions 710-760 may be executed by hardware processor 702 or by a processing engine included in hardware processor 702. Instruction 710 may be executed to create a work plan to delete a backup item stored in a deduplication storage system, where the deduplication storage system stores a plurality of manifests to record an arrival order of a plurality of data units included in the backup item. For example, see Figs. 1 and 4A-4G: Storage controller 110 selects a housekeeping job from job queue 410 and initiates the selected job to delete backup item A. Storage controller 110 loads item A's metadata into storage 115 and reads the manifest identifiers and associated container lists. Storage controller 110 populates the work entries of work map 430 based on item A's metadata 420.For entries in a container list that contain a backreference flag set to a "true" value, the storage controller 110 adds the manifest identifier and the backreference flag to the working entry. For entries in a container list that contain a backreference flag set to a "false" value, the storage controller 110 loads the identified manifest 150 into the storage 115, determines at least one data unit range based on the manifest 150, and then adds the manifest identifier, the backreference flag, and the at least one data unit range to the working entry. Instruction 720 may be executed to select a first work entry of the caretaker work map, where the first work entry identifies a first container index and a first manifest. Instruction 730 may be executed to load the first container index into memory in response to selecting the first work entry, where the first container index includes a backreference data structure. For example, referring to FIGS. 1 and 4A-4G, the storage controller 110 processes the work map 420 by iterating through the work entries. The storage controller 110 selects a first work entry of the work map 420 and reads an identifier for the container index C-2. Further, the storage controller 110 causes the container index C-2 510A to be loaded from the persistent storage 140 into the memory 115. The container index C-2 510A includes the backreference list 530A. Instruction 740 may be executed to identify a backreference entry in the backreference data structure that is indexed by the first manifest. Instruction 750 may be executed to determine, using the backreference entry indexed by the first manifest, a first set of data units contained in the first manifest and indexed in the first container index. Instruction 760 may be executed to decrement a set of reference counts in the first container index for the determined first set of data units. For example, referring to FIGS. 1 and 4A-4G, the memory controller 110 reads the first field in the first working entry (for C-2), thereby obtaining the manifest identifier "MA-5" and a backreference flag set to a true value. The memory controller 110 compares the manifest identifier "MA-5" with the third backreference entry in the backreference list 530A of the container index C-2 510A, thereby obtaining the data unit range R1. The memory controller 110 then decrements the reference counts (in the data unit metadata 520A of the container index C-2 510A) for the data units represented by the data unit range R1. The storage controller 110 then decrements the reference counts (in the data unit metadata 520A of the container index C-2 510A) for the data units represented by the data unit range R1. In another example, the storage controller 110 selects the third working entry (for C-4).Further, the storage controller 110 reads the first field in the third working entry, thereby obtaining the manifest identifier "MA-1," a backreference flag set to a false value, and the data unit range R2. The storage controller 110 then decrements the reference counts (in the data unit metadata 520B of the container index C-4 510B) for the data units represented by the data unit range R2. FIG. 8 - Example procedure for budget management Fig. 8 shows an example process 800 for housekeeping in accordance with some implementations. In some examples, process 800 may be performed using memory controller 110 (shown in Fig. 1). Process 800 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory, computer-readable medium, such as an optical, semiconductor, or magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. Block 810 may include creating, by a storage controller of a deduplication storage system, a housekeeping work map to delete a backup item stored in the deduplication storage system, wherein the deduplication storage system stores a plurality of manifests to record an arrival order of a plurality of data units included in the backup item. Block 820 may include selecting, by the storage controller, a first work entry of the housekeeping work map, wherein the first work entry identifies a first container index and a first manifest. Block 830 may include, in response to selecting the first working entry, loading the first container index into memory, wherein the first container index comprises a backreference data structure. Block 840 may include identifying, in the backreference data structure, a backreference entry indexed by the first manifest. Block 850 may include determining a first set of data items included in the first manifest and indexed by the first container index using the backreference entry indexed by the first manifest. Block 860 may include decrementing a set of reference counts in the first container index for the determined first set of data items. FIG. 9 - Example of a machine-readable data carrier Fig. 9 shows a machine-readable medium 900 on which instructions 910-960 are stored, in accordance with some implementations. Instructions 910-960 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. Machine-readable medium 900 may be a non-transitory storage medium, such as an optical, semiconductor, or magnetic storage medium. Instruction 910 may be executed to generate a housekeeping work map to delete a backup item stored in a deduplication storage system, wherein the deduplication storage system stores a plurality of manifests to record an arrival order of a plurality of data units included in the backup item. Instruction 920 may be executed to select a first work entry of the housekeeping work map, wherein the first work entry identifies a first container index and a first manifest. Instruction 930 may be executed to load the first container index into memory in response to a selection of the first working entry, the first container index comprising a backreference data structure. Instruction 940 may be executed to identify a backreference entry in the backreference data structure that is indexed by the first manifest. Instruction 950 may be executed to determine, using the backreference entry indexed by the first manifest, a first set of data units included in the first manifest that are indexed in the first container index. Instruction 960 may be executed to decrement a set of reference counts in the first container index for the determined first set of data units. In accordance with some implementations of the present disclosure, a controller of a deduplication storage system may generate a housekeeping work map for performing a housekeeping job. The controller may select a work item for processing and load the container index identified in the work item from persistent storage into storage. The controller may read a backreference data structure contained in the container index. Each entry of the backreference data structure may specify a different manifest and data unit range. The data unit range may identify a set of data units contained in the specified manifest and indexed by the container index. The controller may select the backreference entry that specifies the same manifest identified in the work item and extract the data unit range specified in the selected backreference entry.The controller can then decrement the reference counts in the container index corresponding to the extracted data unit range. This allows the work item to be processed without loading the manifest into memory. Accordingly, some implementations can reduce the I / O bandwidth used for housekeeping jobs, thereby improving storage system performance. Note that while Figures 1-9 show various examples, implementations are not limited in this regard. For example, the storage system 100 as shown in Figure 1 may include additional devices and / or components, fewer components, different components, different arrangements, etc. Another example is that the functionality of the storage controller 110 described above may be included in a different machine or software of the storage system 100. Other combinations and / or variations are also possible. Data and instructions are stored in corresponding storage devices embodied as one or more computer-readable or machine-readable storage media. Storage media includes various forms of non-transitory memory, including semiconductor memory such as dynamic or static random-access memories (DRAMs or SRAMs), erasable and programmable read-only memories (EPROMs), electrically erasable and programmable read-only memories (EEPROMs), and flash memory; magnetic disks such as fixed, floppy, and removable disks; other magnetic media including tapes; optical media such as compact discs (CDs) or digital video discs (DVDs), or other types of storage devices. It should be noted that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed throughout a large system with possibly multiple nodes. Such computer-readable or machine-readable storage medium or media are considered part of an article (or article of manufacture). An article or article of manufacture may refer to each individual component or components manufactured. The storage medium or media may be located either in the machine on which the machine-readable instructions are executed or at a remote location from which machine-readable instructions may be downloaded over a network for execution. In the foregoing description, numerous details are set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations from the details described above. The appended claims are intended to cover such modifications and variations.
Claims
A computing device comprising: a processor; a memory; and a machine-readable memory storing instructions, the instructions executable by the processor to: generate a housekeeping work map for deleting a backup item stored in a deduplication storage system, the deduplication storage system storing a plurality of manifests to record an arrival order of a plurality of data units contained in the backup item; select a first work entry of the housekeeping work map, the first work entry identifying a first container index and a first manifest; in response to selecting the first work entry, load the first container index into memory, the first container index comprising a backreference data structure; identify, in the backreference data structure, a backreference entry indexed to the first manifest;Determining a first set of data units included in the first manifest and indexed in the first container index using the backreference entry indexed by the first manifest; andDecrementing a set of reference counts for the determined first set of data units in the first container index.; The computing device of claim 1, including instructions executable by the processor to:determine whether a first flag included in the first working entry indicates that a combination of the first container index and the first manifest is to be processed using the backreference data structure; andidentify the backreference entry indexed by the first manifest in the backreference data structure in response to determining that the first flag indicates that the combination of the first container index and the first manifest is to be processed using the backreference data structure. The computing device of claim 2, including instructions executable by the processor to:select a second work entry of the housekeeping work card, the second work entry identifying a second container index and a second manifest;determine whether a second flag included in the second work entry indicates that a combination of the second container index and the second manifest is to be processed using a second backreference data structure in the second container index;in response to determining that the second flag does not indicate that the combination of the second container index and the second manifest is to be processed using the second backreference data structure in the second container index: determining a second set of data units included in the second manifest and indexed in the second container index based on data unit range information included in the second work entry; and decrementing a second set of reference counts for the determined second set of data units in the second container index.; The computing device of claim 1, including instructions executable by the processor to:initiate a housekeeping job to delete the backup item;identify a set of container indexes affected by the housekeeping job, wherein the set of container indexes includes a third container index and wherein the first manifest references the third container index;determine whether the housekeeping work card includes an existing work entry identifying the third container index; andin response to determining that the housekeeping card includes the existing work entry identifying the third container index, add the identifier of the first manifest to the existing work entry. The computing device of claim 4, including instructions executable by the processor to:determine, based on article metadata representing the backup article, whether to process a combination of the third container index and the first manifest using a third backreference data structure in the third container index;in response to determining that the combination of the third container index and the first manifest should not be processed using the third backreference data structure:load the first manifest into memory from persistent storage;determine, based on the first manifest loaded into memory, a first data unit range identifying a third set of data units included in the first manifest and indexed in the third container index;andadd a reference to the first data unit range to the existing work entry.; The computing device of claim 5, including instructions executable by the processor to:in response to determining that the combination of the third container index and the first manifest is not to be processed using the third backreference data structure, add a third flag to the existing work entry, the third flag indicating that the combination of the third container index and the first manifest is not to be processed using the third backreference data structure. The computing device of claim 5, including instructions executable by the processor to:in response to determining that the combination of the third container index and the first manifest is to be processed using the third backreference data structure, add a fourth flag to the existing work entry, the fourth flag indicating that the combination of the third container index and the first manifest is to be processed using the third backreference data structure. The computing device of claim 4, including instructions executable by the processor to:in response to determining that the housekeeping map does not include the existing work entry identifying the third container index:generate a new work entry identifying the third container index in the housekeeping work map; andadd an identifier of the first manifest to the new work entry. A method comprising: generating, by a storage controller of a deduplication storage system, a housekeeping work map to delete a backup item stored in a deduplication storage system, the deduplication storage system storing a plurality of manifests to record an arrival order of a plurality of data units included in the backup item; selecting, by the storage controller, a first work entry of the housekeeping work map, the first work entry identifying a first container index and a first manifest; in response to selecting the first work entry, loading the first container index into memory, the first container index comprising a backreference data structure; identifying a backreference entry in the backreference data structure indexed by the first manifest;Determining a first set of data units included in the first manifest and indexed in the first container index using the backreference entry indexed by the first manifest; andDecrementing a set of reference counts for the determined first set of data units in the first container index.; The method of claim 9, comprising:determining whether a first flag included in the first working entry indicates that a combination of the first container index and the first manifest is to be processed using the backreference data structure; andidentifying the backreference entry indexed by the first manifest in the backreference data structure in response to determining that the first flag indicates that the combination of the first container index and the first manifest is to be processed using the backreference data structure.The method of claim 10, comprising:selecting a second work entry of the housekeeping work card, the second work entry identifying a second container index and a second manifest;determining whether a second flag included in the second work entry indicates that a combination of the second container index and the second manifest should be processed using a second backreference data structure in the second container index;in response to determining that the second flag does not indicate that the combination of the second container index and the second manifest should be processed using the second backreference data structure in the second container index:determining a second set of data units included in the second manifest and indexed in the second container index based on data unit range information included in the second work entry;anddecrementing a second set of reference counts for the determined second set of data units in the second container index; The method of claim 9, comprising:initiating a housekeeping job to delete the backup item;identifying a set of container indexes affected by the housekeeping job, wherein the set of container indexes includes a third container index, and wherein the first manifest references the third container index;determining whether the housekeeping work card contains an existing work entry identifying the third container index; andin response to determining that the housekeeping card contains the existing work entry identifying the third container index, adding the identifier of the first manifest to the existing work entry. The method of claim 12, comprising:determining, based on the article metadata representing the backup article, whether a combination of the third container index and the first manifest should be processed using a third backreference data structure in the third container index;in response to determining that the combination of the third container index and the first manifest should not be processed using the third backreference data structure:loading the first manifest from persistent storage into memory;determining, based on the first manifest loaded into memory, a first data unit range identifying a third set of data units included in the first manifest and indexed in the third container index; andadding an indication of the first data unit range to the existing working entry. The method of claim 12, comprising:in response to determining that the housekeeping map does not include the existing work entry identifying the third container index:creating a new work entry identifying the third container index in the housekeeping work summary; andadding an identifier of the first manifest to the new work entry. A non-transitory, machine-readable medium storing instructions that, when executed, cause a processor to:create a housekeeping work map to delete a backup item stored in a deduplication storage system, the deduplication storage system storing a plurality of manifests to record an arrival order of a plurality of data units contained in the backup item;select a first work entry of the housekeeping work map, the first work entry identifying a first container index and a first manifest;in response to selecting the first work entry, load the first container index into memory, the first container index comprising a backreference data structure;identify in the backreference data structure a backreference entry indexed to the first manifest;Determining a first set of data units included in the first manifest and indexed in the first container index using the backreference entry indexed by the first manifest; andDecrementing a set of reference counts for the determined first set of data units in the first container index.; The non-transitory machine-readable medium of claim 15, including instructions that, when executed, cause the processor to:determine whether a first flag included in the first working entry indicates that a combination of the first container index and the first manifest is to be processed using the backreference data structure; andidentify the backreference entry indexed by the first manifest in the backreference data structure in response to determining that the first flag indicates that the combination of the first container index and the first manifest is to be processed using the backreference data structure. The non-transitory machine-readable medium of claim 16, including instructions that, when executed, cause the processor to:select a second work entry of the housekeeping work card, the second work entry identifying a second container index and a second manifest;determine whether a second flag included in the second work entry indicates that a combination of the second container index and the second manifest is to be processed using a second backreference data structure in the second container index;in response to determining that the second flag does not indicate that the combination of the second container index and the second manifest is to be processed using the second backreference data structure in the second container index: determining a second set of data units included in the second manifest and indexed in the second container index based on data unit range information included in the second work entry; and decrementing a second set of reference counts for the determined second set of data units in the second container index.; The non-transitory machine-readable medium of claim 15, including instructions that, when executed, cause the processor to:initiate a housekeeping job to delete the backup item;identify a set of container indexes affected by the housekeeping job, wherein the set of container indexes includes a third container index, and wherein the first manifest references the third container index;determine whether the housekeeping work card includes an existing work entry identifying the third container index; andin response to determining that the housekeeping card includes the existing work entry identifying the third container index, add the identifier of the first manifest to the existing work entry. The non-transitory machine-readable medium of claim 18, including instructions that, when executed, cause the processor to:determine, based on article metadata representing the backing article, whether to process a combination of the third container index and the first manifest using a third backreference data structure in the third container index;in response to determining that the combination of the third container index and the first manifest is not to be processed using the third backreference data structure:load the first manifest into memory from persistent storage;determine, based on the first manifest loaded into memory, a first data unit range that identifies a third set of data units included in the first manifest and indexed in the third container index;andadd a reference to the first data unit range to the existing work entry.; The non-transitory machine-readable medium of claim 18, including instructions that, when executed, cause the processor to:in response to determining that the housekeeping map does not include the existing work entry identifying the third container index:create a new work entry identifying the third container index in the housekeeping work map; andadd an identifier of the first manifest to the new work entry.