Reverse reference data structure for a deduplication storage system

Abstract

The present disclosure relates to a reverse reference data structure for a deduplication storage system. Example embodiments relate to deduplication operations in a storage system. Examples include: generating a housekeeping work map to delete backup items stored in a deduplication storage system; selecting a first work entry of the housekeeping work map, wherein the first work entry identifies a first container index and a first manifest; in response to the selection of the first work entry, loading the first container index into the memory, the first container index including a reverse reference data structure; identifying, in the reverse reference data structure, a reverse reference entry indexed to the first manifest; using the reverse reference entry indexed to the first manifest to determine a first set of data units included in the first manifest and indexed in the first container index; and decrementing a set of reference counts of the determined first set of data units in the first container index.

Description

Background Art

[0001] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. Example data reduction techniques include data deduplication. Data deduplication identifies duplicate data units and seeks to reduce or eliminate the number of instances of duplicate data units stored in the storage system. Brief Description of the Drawings

[0002] Some embodiments are described with reference to the following drawings.

[0003] Figure 1 is a schematic diagram of an example storage system in accordance with some embodiments.

[0004] Figure 2 is an illustration of an example data structure in accordance with some embodiments.

[0005] Figures 3A to 3B is an illustration of an example process in accordance with some embodiments.

[0006] Figures 4A to 4G is an illustration of an example operation in accordance with some embodiments.

[0007] Figures 5A to 5E is an illustration of an example operation in accordance with some embodiments.

[0008] Figure 6 is an illustration of an example process in accordance with some embodiments.

[0009] Figure 7 is a schematic diagram of an example computing device in accordance with some embodiments.

[0010] Figure 8 is an illustration of an example process in accordance with some embodiments.

[0011] Figure 9 is a diagram of an example machine-readable medium storing instructions in accordance with some embodiments.

[0012] In all of the drawings, the same reference numerals refer to similar but not necessarily identical elements. The drawings are not necessarily to scale, and the dimensions of some parts may be enlarged to more clearly illustrate the example shown. Additionally, the drawings provide examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments provided in the drawings. Detailed Description

[0013] In this disclosure, unless the context clearly dictates otherwise, the terms "a," "an," or "the" are intended to include the plural forms as well. Similarly, as used in this disclosure, the terms "includes," "including," "comprises," or "comprising" specify the presence of the element, but do not preclude the presence or addition of other elements.

[0014] In some examples, a storage system can backup a data collection (referred to herein as a "stream" or "data stream" of data) in a deduplicated form, thereby reducing the amount of storage space required to store the data stream. The storage system can create a "backup item" to represent the data stream in a deduplicated form. The storage system can perform a deduplication process that includes breaking the data stream into discrete data units (or "chunks") and determining the "fingerprint" (as described below) of these incoming data units. Further, the storage system can compare the fingerprint of the incoming data units with the fingerprints of the stored data units and can thereby determine which incoming data units are replicas of previously stored data units (e.g., when the comparison indicates matching fingerprints). In the case where a data unit is a replica, the storage system can store a reference to the previously stored data unit instead of storing the replicated incoming data unit.

[0015] As used herein, a "fingerprint" refers to a value obtained by applying a function to the content of a data unit (where "content" can include all or a subset of the content of the data unit). Examples of functions that can be applied include hash functions that produce a hash value based on the content of the incoming data unit. Examples of hash functions include cryptographic hash functions such as the Secure Hash Algorithm 2 (SHA-2) hash function (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions can be employed.

[0016] A "storage system" can include a storage device or an array of storage devices. The storage system can also include one or more storage controllers that manage access to the (multiple) storage devices. A "data unit" can refer to any portion of data that can be individually identified within the storage system. In some cases, a data unit can refer to a chunk, a collection of chunks, or any other portion of data. In some examples, the storage system can store data units in persistent storage. Persistent storage can be implemented using one or more (multiple) persistent (e.g., non-volatile) storage devices such as (multiple) disk-based storage devices (e.g., (multiple) hard disk drives (HDDs)), (multiple) solid-state devices (SSDs) (such as (multiple) flash storage devices), etc., or a combination thereof.

[0017] "Controller" may refer to a hardware processing circuit, which may include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuits. Alternatively, "controller" may refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) executable on the hardware processing circuit.

[0018] In some examples, the deduplication storage system may use metadata to process an inbound data stream (e.g., a backup item). For example, such metadata may include data recipes (also referred to herein as "manifests") that specify the order of receipt of specific data units for each backup item. Further, such metadata may include item metadata to represent each received backup item (e.g., data stream) in a deduplicated form. The item metadata may include an identifier of a set of manifests and may indicate the sequential order of the set of manifests. Processing each backup item may be referred to herein as a "backup process". Subsequently, in response to a read request, the deduplication system may use the item metadata and the set of manifests to determine the order of receipt of the data units and may thereby recreate the original data stream of the backup item. Accordingly, the set of manifests may be a representation of the original backup item. A manifest may include a series of records, each record representing a particular set of data units. The records in a manifest may include one or more fields identifying a container index that indexes the data units (e.g., includes storage information of the data units). For example, the container index may include one or more fields that specify location information (e.g., container, offset, etc.) of the stored data units, compression and / or encryption characteristics of the stored data units, etc. Further, the container index may include a reference count indicating the number of manifests that reference each data unit.

[0019] In some examples, when a data unit (e.g., in a data stream) is received, it can be matched with one or more container indexes to determine whether the same chunk has already been stored in a container of the deduplication storage system. For example, the deduplication storage system can compare the fingerprint of the received data unit with the fingerprints in one or more container indexes. If no matching fingerprint is found in the searched container indexes, the received data unit can be added to a container, and an entry for the received data unit can be added to the container index corresponding to that container. However, if a matching fingerprint is found in the searched container indexes, it can be determined that a data unit identical to the received data unit has already been stored in a container. In response to that determination, the reference count of the corresponding entry is incremented, and the received data unit is not stored in the container (since it already exists in one of the containers), thereby avoiding storing duplicate data units in the deduplication storage system. As used herein, the term "matching operation" can refer to an operation for comparing the fingerprints of a collection of multiple data units (e.g., from a particular backup data stream) with the fingerprints stored in a container index.

[0020] In some examples, the deduplication storage system can perform a housekeeping job or process to maintain the accuracy of the included metadata. For example, when a data unit is deleted from a given manifest (e.g., due to changes to the data stream or item represented by the manifest), the housekeeping job can include decrementing the reference count of the data unit by one (i.e., indicating that there is one less manifest referencing the data unit). In some examples, the housekeeping job for deleting a backup item can include identifying the manifests included in the backup item, sequentially accessing and loading them into memory, and reading the manifests to determine the container identifiers and address information of the data units in each manifest. The housekeeping job can also include accessing the container index and loading it into memory (e.g., using the container identifier), decrementing the reference count of the data unit (e.g., using the address information of the data unit), and then saving the container index to persistent storage. Accordingly, performing such a housekeeping job may involve a relatively large number of input / output (I / O) operations to load multiple metadata items (e.g., manifests and container indexes) from persistent storage into memory and write at least some of the multiple metadata items from memory to persistent storage. Accordingly, performing such a housekeeping job may degrade the performance of the deduplication storage system.

[0021] According to some embodiments of the present disclosure, a controller of a deduplication storage system can generate a housekeeping work map for performing housekeeping jobs (e.g., to delete backup items that are no longer needed). Entries of the work map (also referred to as "work entries") can identify container indices and manifests. The controller can select work entries for processing and can then load the container indices identified in the work entries from persistent storage into memory. The controller can access or read a back-reference data structure included in the container index. Each entry of the back-reference data structure (also referred to as a "back-reference entry") can specify a different manifest and a range of data units. The range of data units can identify a set of data units included in the specified manifest and indexed by the container index. The controller can select back-reference entries that specify the same manifest identified in the work entry and can extract the range of data units specified in the selected back-reference entries. The controller can then decrement the reference count corresponding to the extracted range of data units in the container index. In this way, work entries can be processed without loading the manifest into memory. Accordingly, some embodiments can reduce the amount of I / O bandwidth used for housekeeping jobs and can thereby improve the performance of the storage system. Further aspects of the disclosed housekeeping process are discussed below with reference to Figures 1 to 9 further aspects of the disclosed housekeeping process.

[0022] Figure 1 - Example storage system

[0023] Figure 1 An example of a storage system 100 including a storage controller 110, a memory 115, and a persistent storage 140 is shown in accordance with some embodiments. The persistent storage 140 can include one or more non-transitory storage media such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc., or a combination thereof. The memory 115 can be implemented with a semiconductor memory such as random access memory (RAM). In some examples, the storage controller 110 can be implemented via hardware (e.g., electronic circuitry) or a combination of hardware and programming (e.g., including at least one processor and instructions executable by the at least one processor and stored on at least one machine-readable storage medium).

[0024] As Figure 1 shown, the memory 115 and the persistent storage 140 can store various data structures, which at least include a work map 120, item metadata 130, a manifest 150, a container index 160, and a data container 170. In some examples, copies of the manifest 150, the container index 160, and the data container 170 can be transferred between the memory 115 and the persistent storage 140 (e.g., via read and write input / output (I / O) operations).

[0025] In some embodiments, the storage system 100 may perform deduplication on the stored data. For example, the storage controller 110 may receive an inbound data stream 105 (also referred to herein as a “backup item”) that includes multiple data units, and may store at least one copy of each data unit in a data container 170 (e.g., by appending the data unit to the end of the data container 170). In some examples, each data container 170 may be divided into entities 175, where each entity 175 includes multiple stored data units.

[0026] In one or more embodiments, the storage controller 110 may generate a fingerprint for each received data unit. For example, the fingerprint may include a full or partial hash value based on the data unit. To determine whether an incoming data unit is a replica of a stored data unit, the storage controller 110 may compare the fingerprint generated for the incoming data unit with the fingerprints in at least one container index. If a match is identified, the storage controller 110 may determine that the storage system 100 has already stored a replica of the incoming data unit. The storage controller 110 may then store a reference to the previous data unit instead of storing the replicated incoming data unit.

[0027] In some embodiments, the storage controller 110 may generate item metadata 130 to represent each received backup item (e.g., data stream 105) in a deduplicated form. Each item metadata 130 may include identifiers for a set of manifests 150 and may indicate the sequential order of the set of manifests 150. The manifests 150 record the order in which data units are received. Further, the manifests 150 may include pointers or other information indicating the container index 160 that indexes each data unit. In some embodiments, the container index 160 may indicate the storage location of the data unit. For example, the container index 160 may include information specifying that the data unit is stored at a particular offset within an entity and that the entity is stored at a particular offset within the data container 170. Further, the container index 160 may include a reference count indicating the number of manifests 150 that reference each data unit.

[0028] In some embodiments, the storage controller 110 may receive a read request to access stored data and, in response, may access the project metadata 130 and the manifest 150 to determine the sequence of data units that constitute the original data. The storage controller 110 may then use the pointer data included in the manifest 150 to identify the container index 160 that indexes the data unit. Further, the storage controller 110 may use the information included in the identified container index 160 (as well as the information included in the manifest 150) to determine the storage location of the data unit (e.g., data container 170, entity 175, offset, etc.), and may then read the data unit from the determined location.

[0029] In some embodiments, the storage controller 110 may use the work map 120 to perform housekeeping jobs for the storage system 100. For example, the storage controller 110 may receive a request to delete a particular backup item and, in response, may schedule a housekeeping job to be performed at a later time. When initiating the housekeeping job, the storage controller 110 may load the item metadata 130 for the backup item from the persistent storage 140 into the memory 115. In some embodiments, the storage controller 110 may use the item metadata 130 to identify a manifest 150 representing the backup item and identify a container index 160 referenced by the identified manifest 150. The storage controller 110 may populate the work map 120 with a set of work entries corresponding to the container index 160 identified in the item metadata 130. Each work entry may identify a corresponding container index 160 and may also identify a set of manifests 150, each manifest referencing a corresponding container index 160. Figures 4A to 5E Some example implementations of the working map 120 and the project metadata 130 are discussed.

[0030] In some embodiments, the storage controller 110 may select a work item of the work map 120 for processing and may then load the container index 160 identified in the work item from the persistent storage 140 into the memory 115. The container index 160 may include a data structure for storing a set of back-reference entries, where each back-reference entry specifies a different manifest 150 and a data unit range (e.g., information for identifying a set of data units included in the manifest 150 and indexed by the container index 160). The storage controller 110 may select the back-reference entries that specify the same manifest 150 identified in the work item and may extract the data unit ranges specified in the selected back-reference entries. The data unit ranges (extracted from the back-reference entries) may be used to deterministically identify one or more data units for which the reference count is to be decremented for a housekeeping job. The storage controller 110 may then decrement the reference counts corresponding to the one or more data units identified by the extracted data unit ranges in the container index 160. In this way, using the back-reference entries in the container index 160 may allow for decrementing the appropriate reference counts for a housekeeping job without having to load the manifest 150 into the memory to identify the one or more data units associated with those reference counts. Accordingly, some embodiments may reduce the amount of I / O bandwidth used for housekeeping jobs and may thereby improve the performance of the deduplication storage system 100.

[0031] In some embodiments, the storage controller 110 may use the item metadata 130 to obtain a Boolean flag (also referred to as a “back-reference flag” or “BR flag”) for each combination of the manifest 150 and the container index 160 identified in the item metadata 130 (also referred to as a “manifest-CI combination”), the Boolean flag indicating whether the back-reference entries included in the container index 160 will be used to process the particular manifest-CI combination. The storage controller 110 may populate the work items of the work map 120 based on the back-reference flags associated with each manifest. For example, if the back-reference flag is set to a “false” value (i.e., indicating that back-reference entries are not used to process the manifest-CI combination), the storage controller 110 loads the manifest 150 into the memory 115 to determine the data unit ranges corresponding to the reference counts to be decremented in the container index 160 and then appends the determined data unit ranges and the identifier of the manifest 150 to the work item (e.g., in a data field). Otherwise, if the back-reference flag is set to a “true” value (i.e., indicating that back-reference entries will be used to process the manifest-CI combination), the storage controller 110 appends the identifier of the manifest 150 to the work item but does not load the manifest 150 into the memory 115 and does not append the data unit ranges to the work item. Referring below to Figures 3A to 3BFurther discuss an example process for performing housekeeping using a reverse reference data structure.

[0032] Figure 2 - Example data structure

[0033] Now refer to Figure 2 , which shows an illustration of an example data structure 200 used in deduplication. As shown, the data structure 200 may include item metadata 202, a manifest 203, a container index 220, and a data container 250. In some examples, the item metadata 202, the manifest 203, the container index 220, and the data container 250 may generally correspond to example embodiments of the item metadata 130, the manifest 150, the container index 160, and the data container 170 (as Figure 1 shown). In some examples, the data structure 200 may be generated and / or managed by a storage controller 110 (as Figure 1 shown).

[0034] In some embodiments, the item metadata 202 may include a plurality of manifest identifiers 205. Each manifest identifier 205 may identify a different manifest 203. In some embodiments, the manifest identifiers 205 may be arranged in a stream order (i.e., based on the order of receipt of the data units represented by the identified manifest 203). Further, the item metadata 202 may include a container list 204 associated with each manifest identifier 205. In some embodiments, the container list

[0035] 204 may include identifiers of a set of container indexes 220 that index the data units to be included in the associated manifest 203 (i.e., the manifest

[0036] 203 identified by the associated manifest identifier 205).

[0037] Further, in some embodiments, the container list 204 may include a reverse reference flag associated with each container index identifier. The reverse reference flag may indicate whether the reverse reference list 222 (discussed below) included in the container index 220 will be used to process the manifest-CI combination (corresponding to the associated container index identifier and manifest identifier 205). Refer to Figures 4A to 4G below for an example embodiment of the container list 204.

[0038] Although for simplicity <id= Figure 2In the illustration, only one of each data structure is shown, but data structure 200 may include multiple instances of item metadata 202, each instance including or pointing to one or more manifests 203. In such an example, data structure 200 may include multiple manifests 203. A manifest 203 may reference multiple container indices 220, each container index corresponding to one of multiple data containers 250. Each container index 220 may include one or more data unit records 230 and one or more entity records 240.

[0039] As Figure 2 shown, in some examples, each manifest 203 may include one or more manifest records 210. Each manifest record 210 may include various fields such as an offset, a length, a container index, and a cell address. In some embodiments, each container index 220 may include any number of (multiple) data unit records 230 and (multiple) entity records 240. Each data unit record 230 may include various fields such as a fingerprint (e.g., a hash of the data unit), a cell address, an entity identifier, a cell offset (i.e., the offset of the data unit within the entity), a reference count value, and a cell length. In some examples, the reference count value may indicate the number of manifest records 210 that reference the data unit record 230. Further, each entity record 240 may include various fields such as an entity identifier, an entity offset (i.e., the offset of the entity within the container), a storage length (i.e., the length of the data unit within the entity), a decompressed length, a checksum value, and compression / encryption information (e.g., a compression type, an encryption type, etc.). In some embodiments, each container 250 may include any number of entities 260, and each entity 260 may include any number of stored data units.

[0040] In some embodiments, the cell address (included in the manifest record 210 and the data unit record 230) may be an identifier that deterministically identifies a particular data unit within a given container index 220. In some examples, the cell address may be a numerical value (referred to as an "arrival number") that indicates the arrival order sequence (also referred to as the "ingest order") of the data unit being indexed in a given container index 220 (e.g., when receiving an inbound data stream and deduplicating it). For example, an arrival number "1" may be assigned to the first data unit to be indexed in container index 220 (e.g., by creating a new data unit record 230 for the first data unit), an arrival number "2" may be assigned to the second data unit, an arrival number "3" may be assigned to the third data unit, and so on. However, other embodiments are possible.

[0041] In some embodiments, the manifest record 210 may use a run - length reference format to represent a contiguous range (e.g., a portion of a data stream) of data units indexed within a single container index 220. The run - length reference may be recorded in the unit address field and the length field of the manifest record 210. For example, the unit address field may indicate the arrival number of the first data unit in the represented data unit range, and the length field may indicate the number N (where "N" is an integer) of data units that follow the data unit specified by the arrival number in the unit address field within the data unit range. The data units in the data unit range may have consecutive arrival numbers (e.g., because they are consecutive in the ingested data stream). Thus, the data unit range may be represented by the arrival number of the first data unit in the data unit range (e.g., specified in the unit address field of the manifest record 210) and the number N of additional data units in the data unit range (e.g., specified in the length field of the manifest record 210). The additional data units that follow the first data unit in the data unit range may be deterministically obtained by calculating N arrival numbers that sequentially follow the specified arrival number of the first data unit, where these N arrival numbers identify the additional data units in the data unit range. In such an example, the manifest record 210 may include the arrival number "X" in the unit address field and the number N in the length field to indicate a data unit range that includes the data unit specified by arrival number X and the data units specified by arrival number X + i, where i = 0 to i = N (including 0 and N) (where "i" is an integer). In this way, the manifest record 210 can be used to identify all data units in the data unit range.

[0042] In one or more embodiments, the data structure 200 may be used to retrieve stored deduplicated data. For example, a read request may specify an offset and a length of data within a given file. These request parameters may be matched with the offset and length fields of a particular manifest record 210. Then, the container index and unit address of the particular manifest record 210 may be matched with a particular data unit record 230 included in the container index 220. Further, the entity identifier of the particular data unit record 230 may be matched with the entity identifier of a particular entity record 240. Additionally, one or more other fields (e.g., entity offset, storage length, checksum, etc.) of the particular entity record 240 may be used to identify the container 250 and the entity 260, and then the data unit may be read from the identified container 250 and entity 260.

[0043] In some embodiments, each container index 220 may include a reverse reference list 222. The reverse reference list 222 may be a data structure for storing a set of reverse reference entries, where each reverse reference entry specifies a different manifest 203 and at least one data unit range (e.g., a set of one or more data units included in the manifest 203 and indexed by the container index 220).

[0044] In some embodiments, a reverse reference entry (in the reverse reference list 222) may specify the data unit range as a run - length reference that identifies a first data unit and the length of the data unit range in the container index 220. For example, the run - length reference may use a first arrival number "X" assigned to the first data unit in the container index 220 to identify the first data unit in the data unit range. Further, the run - length reference may identify a length value N that indicates the number of arrival numbers assigned to the remaining data units in the data unit range that sequentially follow the first arrival number "X" in the container index 220. In such an example, the run - length reference included in the reverse reference entry can be used to identify the first data unit in the range (with the first arrival number "X"), and then identify the remaining data units in the range (with N arrival numbers that coherently follow the first arrival number "X"). In this way, the run - length reference included in the reverse reference entry can be used to deterministically identify all data units in the data unit range.

[0045] In other embodiments, the reverse reference entry may use other identifiers or representations to specify the data unit range (including one or more data units). For example, the reverse reference entry may specify the data unit range as the fingerprint of a single data unit, a set of fingerprints of multiple data units, a set of arrival numbers of multiple data units, an offset (e.g., byte distance from the end), an offset and a length, etc.

[0046] In some embodiments, when initiating a housekeeping job to delete a backup item, the storage controller (e.g., Figure 1The storage controller 110 shown can load the item metadata 202 of a backup item into the memory. The storage controller can use the item metadata 202 to identify a specific manifest 203 representing the backup item and to identify at least one container index 220 referenced by the identified manifest 203. The storage controller can load the identified container index 220 into the memory and can then access the reverse reference list 222 included in the identified container index 220. Further, the storage controller can identify a reverse reference entry (in the reverse reference list 222) that specifies the specific manifest 203 and can read the data unit range information (e.g., run-length reference) included in the identified reverse reference entry. The storage controller can then decrement the reference count corresponding to the data units identified by the data unit range information in the container index 220. In this way, using the reverse reference list 222 in the container index 220 can allow the appropriate reference counts to be decremented for housekeeping operations without having to load the specific manifest 203 into the memory to identify the one or more data units associated with those reference counts. Accordingly, some embodiments can reduce the amount of I / O bandwidth used for housekeeping operations and can thereby improve the performance of the deduplication storage system.

[0047] In some embodiments, each time a container index 220 is generated or updated to include information about a specific manifest 203, the reverse reference list 222 in the container index 220 is updated to identify the manifest 203 and to identify at least one data unit range of the data units of the manifest 203 indexed by the container index 220. Further, when the container index 220 is no longer associated with the manifest 203, the reverse reference entry for the manifest 203 can be removed from the reverse reference list 222. The following refers to Figures 5A to 5E Describe an example embodiment of the reverse reference list 222.

[0048] Figure 3A and 4A to Figure 4G - Example process for generating working drawings

[0049] Figure 3A An example process 300 for generating a housekeeping work map according to some embodiments is shown. For illustrative purposes, the following may refer to Figures 4A to 4G Describe the details of process 300. However, other embodiments are also possible. In some examples, the storage controller 110 can be used (as Figure 1Process 300 is performed as shown. Process 300 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 300 can be executed by a single processing thread. In other embodiments, process 300 can be executed in parallel by multiple processing threads (e.g., concurrently using a working map and performing multiple housekeeping jobs).

[0050] Block 310 can include scheduling housekeeping jobs in a job queue. Block 312 can include initiating a housekeeping job to delete backup items stored in a deduplicated storage system. For example, referring to Figure 4A , job queue 410 is a data structure for queuing requests to delete backup items. A storage controller (e.g., Figure 1 storage controller 110 therein) receives multiple delete requests to delete backup items A - C stored in the deduplicated storage system. In response to the received requests, the storage controller schedules multiple housekeeping jobs in job queue 410 to delete backup items A - C. The storage controller selects a housekeeping job from job queue 410 and initiates the selected job to delete backup item A.

[0051] Block 314 can include loading item metadata into a memory. For example, referring to Figure 1 and Figures 4A to 4B , storage controller 110 loads item A metadata 420 (i.e., the item metadata of backup item A) from persistent storage 140 into memory 115. Item A metadata 420 includes multiple manifest identifiers 422 and multiple container lists 425, where each container list 425 is associated with a different manifest identifier 422. Each container list 425 includes a set of CI entries. Each CI entry includes a CI identifier for each container index 160 that indexes data units included in the manifest 150 identified by the associated manifest identifier 422. In other words, each CI entry represents a different manifest - CI combination, i.e., a combination of manifest 150 (identified by manifest identifier 422) and container index 160 (identified by the CI identifier in the CI entry). Further, each CI entry includes a backward reference flag ("BR flag") that indicates whether the backward reference data structure included in container index 160 will be used to process the manifest - CI combination (represented by the CI entry).

[0052] The block 316 may include selecting a container index (CI) entry from a list of containers included in the item metadata. The block 318 may include determining the container index, manifest, and back-reference (BR) flag of the selected CI entry. For example, referring to Figure 4B , the storage controller accesses the item A metadata 420 to read the manifest identifier MA-1 422 and the associated list of containers 425. Further, the storage controller reads the first CI entry (in the list of containers 425) that includes the container index identifier C-3. Further, the first CI entry includes a back-reference flag set to true ("T"), indicating that the back-reference data structure included in the container index C-3 will be used to process the combination of the manifest MA-1 and the container index C-3.

[0053] The decision block 320 may include determining whether the working diagram already includes a work entry associated with the identified container index (i.e., the container index identified in the CI entry selected at block 316). If so ("yes"), the process 300 may continue at the decision block 324 (described below). Otherwise, if it is determined at the decision block 320 that the working diagram does not include a work entry associated with the identified container index ("no"), the process 300 may continue at the block 322, including creating a new work entry associated with the identified container index. For example, referring to Figure 4B , the storage controller determines that the working diagram 430 does not include a work entry for the container index C-3 (identified in the first CI entry of the list of containers 425). In response to this determination, the storage controller creates a new work entry for C-3 in the working diagram 430 (i.e., a new work entry indexed to the container index C-3). In another example, referring to Figure 4D , the storage controller determines that the working diagram 430 does not include a work entry for the container index C-4 (identified in the second CI entry of the list of containers 425). In response to this determination, the storage controller creates a new work entry for C-4.

[0054] The decision block 324 may include determining whether the back-reference flag in the selected CI entry is set to a true value. If so ("yes"), the process 300 may continue at the block 330, including adding the manifest identifier and the back-reference flag to the work entry of the identified container index. For example, referring to Figure 4C , the storage controller determines that the CI entry for C-3 includes a back-reference flag set to true ("T"). In response to this determination, the storage controller adds or inserts the manifest identifier "MA-1" and the back-reference flag "BR=T" into the first field ("field 1") of the work entry for C-3 (in the working diagram 430).

[0055] However, if at decision block 324 it is determined that the back-reference flag in the selected CI entry is not set to a true value ("No"), then process 300 can continue at block 326, including loading the identified manifest into memory and reading the identified manifest to determine the range of (multiple) data units for the identified container index. Block 328 can include adding the manifest identifier, the address information of the (multiple) data unit ranges, and the back-reference flag to the working entry for the identified container index. For example, referring to Figure 4E , the storage controller determines that the CI entry for C-4 includes a back-reference flag set to false ("F"). In response to this determination, the storage controller loads manifest MA-1 440 from persistent storage 140 into memory 115. The storage controller reads manifest MA-1 440 to determine the range of data units R2 for the reference count to be decremented in container index C-4. The storage controller then adds the manifest identifier "MA-1", the data identifying data unit range R-2, and the back-reference flag "BR = F" to the first field of the working entry for C-4. For example, the data identifying data unit range R-2 can include one or more run-length references, (multiple) data unit fingerprints, (multiple) data unit arrival numbers, etc.

[0056] Decision block 332 can include determining whether the working map is full (e.g., the maximum number of working entries has been reached) or whether there are no more CI entries remaining in the container list of the item metadata. If not ("No"), then process 300 can return to block 316 (e.g., to select another CI entry from the container list of the item metadata). Otherwise, if at decision block 332 it is determined that the working map is full or there are no more CI entries remaining in the container list, then process 300 can continue at block 334, including processing the working map to complete at least a portion of the housekeeping job. An example expansion of block 334 (i.e., an example process for processing the working map) is described below with reference to Figure 3B and Figures 5A to 5E .

[0057] For example, referring to Figure 4F , after creating working entries for container indexes C-3 and C-4, the storage controller determines that the working map 430 already includes the working entry for container index C-2 (identified in the third CI entry of container list 425). In response to this determination, the storage controller adds the manifest identifier "MA-1" and the back-reference flag "BR = T" to the second field ("Field 2") of the existing working entry for C-2. In another example, referring to Figure 4G, the storage controller determines that the working graph 430 does not include the working entry for container index C-7 (identified in the fourth CI entry of the container list 425). In response to this determination, the storage controller creates a new working entry for C-7 and adds the manifest identifier "MA-1" and the reverse reference flag "BR=T" (i.e., true value) to the first field ("Field 2") of the new working entry for C-7.

[0058] Referring again to Figure 3A , after processing the working graph (at block 334), process 300 can continue at decision block 336, including determining whether there are any more CI entries remaining in the container list of the project metadata. If so ("yes"), then process 300 can return to block 316 (e.g., to select another CI entry from the container list of the project metadata). Otherwise, if it is determined at decision block 336 that there are no more CI entries remaining in the container list ("no"), then process 300 can continue at block 338, including deleting the manifest identified in the project metadata. Block 340 can include deleting the project metadata. After block 340, process 300 can return to block 312 (e.g., to initiate another housekeeping job to delete another backup project). For example, referring to Figure 1 and Figure 4G , the storage controller 110 deletes each manifest 150 identified in the project A metadata 420 from the persistent storage 140 and the memory 115. Further, the storage controller 110 deletes the project A metadata 420 from the persistent storage 140 and the memory 115.

[0059] Figure 3B and Figures 5A to 5E - Example process for processing working drawings

[0060] Figure 3B illustrates an example process 334 for processing a housekeeping working graph. Process 334 can illustrate an example expansion of block 334 (as Figure 3A shown). Thus, in some embodiments, process 334 can be performed after determining that the working graph is full or that there are no more CI entries remaining in the container list of the project metadata (at decision block 332 in Figure 3A ).

[0061] For illustration purposes, the following can refer to Figure 1 and Figures 5A to 5E to describe the details of process 334. However, other embodiments are also possible. In some examples, the storage controller 110 can be used ( Figure 1The process 334 is performed as shown. The process 334 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, the process 334 can be executed by a single processing thread. In other embodiments, the process 334 can be executed in parallel by multiple processing threads.

[0062] At block 350, an outer loop (defined by blocks 350 - 354) can be entered to process each work entry in the housekeeping work map. Block 352 can include loading the container index identified in the current work entry into memory. For example, referring to Figure 5A , the storage controller (e.g., Figure 1 the storage controller 110 in

[0063] traverses the work entries of the work map 420. The storage controller selects the first work entry of the work map 420 and reads the identifier of the container index C-2. Then, the storage controller causes the container index C-2 510A to be loaded from the persistent storage 140 into the memory 115. At block 354, an inner loop (defined by blocks 354 - 362) can be entered to process each manifest identified in the current work entry (i.e., processing the work entry in the current iteration of the outer loop). Decision block 356 can include determining whether the back-reference flag associated with the current manifest is set to true. If so ("yes"), the process 334 can continue at block 360, including performing a lookup for the identified manifest in the back-reference list of the loaded container index to identify the data unit range(s). Block 362 can include decrementing the reference count in the loaded container index for the data unit range(s) identified in the back-reference list. After block 362, the process 334 can return to block 354 to perform another iteration of the inner loop (e.g., processing another manifest in the work entry). Further, after all iterations of the inner loop have been completed (e.g., all manifests in the current work entry have been processed), the process 334 can continue at block 364, including saving the container index to the persistent storage. After block 364, the process 334 can return to block 350 to perform another iteration of the outer loop (e.g., processing another work entry included in the work map).

[0064] For example, referring to Figure 5B, the storage controller reads a first field (“Field 1”) in a first work entry (of container index C-2), thereby obtaining a manifest identifier “MA-5” and a back-reference flag “BR=T” (i.e., a true value). The storage controller matches the manifest identifier “MA-5” with a third back-reference entry in the back-reference list 530A of container index C-2 510A and reads data range information in the third back-reference entry to identify a data unit range R1. The storage controller identifies a reference count (in the data unit metadata 520A of container index C-2 510A) corresponding to the data units included in the data unit range R1. The storage controller then decrements the reference count of the data units in the identified data unit range R1. In some examples, the data range information included in the back-reference list 530A may be specified as a run-length reference. In such an example, the storage controller reads the run-length reference (in the back-reference entry) to identify a first data unit in the data unit range R1 (e.g., having a first arrival number “X”), and then identifies the remaining data units in the data unit range R1 (e.g., having N arrival numbers that coherently follow the first arrival number “X”). In this way, the storage controller uses the run-length reference in the back-reference entry to identify all the data units in the data unit range R1, and then decrements the reference count (in the data unit metadata 520A of container index C-2 510A) corresponding to the data units identified in the data unit range R1. In this way, the back-reference list 530A in container index C-2 510A can be used to decrement the appropriate reference counts for housekeeping jobs without having to load the manifest “MA-5” into memory to identify the data units associated with those reference counts.

[0065] In another example, referring to Figure 5C , the storage controller reads a second field (“Field 2”) in a first work entry (of container index C-2), thereby obtaining a manifest identifier “MA-1” and a back-reference flag “BR=T”. The storage controller matches the manifest identifier “MA-1” with a first back-reference entry in the back-reference list 530A, thereby obtaining a data unit range R3. The storage controller then decrements the reference count (in the data unit metadata 520A of container index C-2 510A) of the data units represented by the data unit range R3. Further, after processing the first and second fields of the first work entry (of container index C-2), the storage controller saves container index C-2 510A to persistent storage and then selects a second work entry (of container index C-3) of the working graph 420 for processing.

[0066] Referring again to Figure 3B, if it is determined at decision block 356 that the back-reference flag associated with the current manifest is not set to a true value (“no”), then process 334 can continue at block 358, including decrementing the reference count in the identified container index of the (multiple) data unit ranges listed in the current work entry. After block 358, process 334 can return to block 354 to perform another iteration of the inner loop (e.g., process another manifest in the work entry). For example, referring to Figure 5D , working diagram 430 is illustrated as representing the state after the first work entry (of container index C-2) and the second work entry (of container index C-3) have been processed and completed. The storage controller selects the next work entry of working diagram 420 (i.e., the third entry) and reads the identifier of container index C-4. Then, the storage controller causes container index C-4 510B to be loaded from persistent storage 140 into memory 115. Further, referring to Figure 5E , the storage controller reads the first field in the third work entry (of container index C-4), thereby obtaining the manifest identifier “MA-1”, the back-reference flag “BR = F” (i.e., false value), and the data unit range R2 (e.g., stored at Figure 3A block 328). The storage controller then decrements the reference count (in the data unit metadata 520B of container index C-4 510B) of the data units represented by data unit range R2.

[0067] Referring again to Figure 3B , after all iterations of the outer loop have been completed (e.g., all work entries in the working diagram have been processed), process 334 can be completed. For example, referring to Figure 3A , the storage controller can restart process 300 at decision block 336, including determining whether there are any more CI entries remaining in the container list of the item metadata, and if so, selecting another CI entry for processing by using working diagram 120.

[0068] Figure 6 - Example process for generating metadata

[0069] Figure 6 Illustrated is an example process 600 for generating metadata according to some embodiments. For illustrative purposes, the details of process 600 can be described below with reference to Figure 1 and Figures 4A to 5E which show examples according to some embodiments. However, other embodiments are possible. In some examples, storage controller 110 can be used ( Figure 1Process 500 is performed as shown. Process 600 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.

[0070] Block 610 can include receiving a backup item to be stored in persistent storage of the deduplication storage system. Block 620 can include generating a fingerprint for a data unit of the received backup item. For example, referring to Figure 1 , storage controller 110 receives a backup item (e.g., data stream 105) to be stored in deduplication storage system 100 and generates a fingerprint for the data units in the received backup item.

[0071] Block 630 can include matching the generated fingerprint with fingerprints stored in a container index of the deduplication storage system. Block 640 can include identifying a subset of the generated fingerprints that do not match the fingerprints stored in the container index. Block 650 can include storing the identified subset of the generated fingerprints in a set of container indexes. For example, referring to Figure 1 , storage controller 110 compares the generated fingerprint with the fingerprints included in container index 160. If a match is identified for a data unit, storage controller 110 determines that the storage system 100 has already stored a copy of the data unit. In response to this determination, storage controller 110 stores a reference to the previous data unit (e.g., in manifest 150) in a deduplicated form. Otherwise, if no match is identified for the data unit, storage controller 110 stores the data unit in data container 170 and adds an entry for the data unit to container index 160 corresponding to data container 170.

[0072] Block 660 can include generating a set of manifests to record the order of the data units of the received backup item. Block 670 can include storing a reverse reference list in each container index to record the manifests and data unit ranges that reference the container index. For example, referring to Figure 1 and Figure 5A, the storage controller 110 records the order in which data units are received in one or more manifests 150. Further, the storage controller 110 generates a reverse reference list 445 in the container index C-2 440. The reverse reference list 445 identifies each manifest 150 (e.g., MA-1) that currently references the container index C-2 400. Further, the reverse reference list 445 can also identify, for each identified manifest 150, at least one data unit range that represents a set of data units included in the manifest 150 and indexed by the container index C-2 400. In some embodiments, the storage controller 110 can continuously update the manifest list 445 during operation (e.g., to reflect manifest deletions or additions to the manifest list 445 during changes to the data stored in the storage system 100).

[0073] Block 680 can include generating item metadata to record the identifiers of the set of manifests. Block 690 can include storing at least one container list in the item metadata to record the set of container indexes and associated reverse reference flags. After block 690, process 600 can complete. For example, referring Figure 1 and Figure 4B , the storage controller 110 generates item A metadata 420 to represent backup item A. The item A metadata 420 includes a plurality of manifest identifiers 422 and a plurality of container lists 425, where each container list 425 is associated with a different manifest identifier 422. Each container list 425 includes a set of CI entries. Each CI entry includes a CI identifier for each container index 160 that indexes data units included in the manifest 150 identified by the associated manifest identifier 422. Further, each CI entry includes a reverse reference flag that indicates whether the reverse reference data structure included in the container index 160 will be used to process the manifest-CI combination (represented by the CI entry).

[0074] Figure 7 - Example computing device

[0075] Figure 7 shows a schematic diagram of an example computing device 700. In some examples, the computing device 600 can generally correspond to some or all of the storage system 100 (as Figure 1 shown). As shown, the computing device 700 can include a hardware processor 702, a memory 704, and a machine-readable storage device 705 that includes instructions 710-760. The machine-readable storage device 705 can be a non-transitory medium. The instructions 710-760 can be executed by the hardware processor 702 or by a processing engine included in the hardware processor 702.

[0076] Instruction 710 can be executed to generate a housekeeping work map to delete backup items stored in a deduplication storage system, where the deduplication storage system stores multiple manifests to record the arrival order of multiple data units included in the backup items. For example, referring to Figure 1 and Figures 4A to 4G , the storage controller 110 selects a housekeeping job from the job queue 410 and starts the selected job to delete backup item A. The storage controller 110 loads the item A metadata 420 into the memory 115 and reads the manifest identifier and the associated container list. The storage controller 110 populates the work entries of the work map 430 based on the item A metadata 420. For an entry in the container list that includes a reverse reference flag set to a "true" value, the storage controller 110 adds the manifest identifier and the reverse reference flag to the work entry. Further, for an entry in the container list that includes a reverse reference flag set to a "false" value, the storage controller 110 loads the identified manifest 150 into the memory 115, determines at least one data unit range based on the manifest 150, and then adds the manifest identifier, the reverse reference flag, and at least one data unit range to the work entry.

[0077] Instruction 720 can be executed to select a first work entry of the housekeeping work map, where the first work entry identifies a first container index and a first manifest. Instruction 730 can be executed to load the first container index into the memory in response to the selection of the first work entry, and the first container index includes a reverse reference data structure. For example, referring to Figure 1 and 4A to Figure 4G , the storage controller 110 processes the work map 420 by traversing the work entries. The storage controller 110 selects the first work entry of the work map 420 and reads the identifier of the container index C-2. Further, the storage controller 110 causes the container index C-2 510A to be loaded from the persistent storage 140 into the memory 115. The container index C-2 510A includes a reverse reference list 530A.

[0078] Instruction 740 can be executed to identify a reverse reference entry indexed to the first manifest in the reverse reference data structure. Instruction 750 can be executed to use the reverse reference entry indexed to the first manifest to determine a first set of data units included in the first manifest and indexed in the first container index. Instruction 760 can be executed to decrement a set of reference counts of the determined first set of data units in the first container index.

[0079] For example, referring to Figure 1 and 4A to Figure 4G, the storage controller 110 reads the first field in the first working entry (of C-2), thereby obtaining the manifest identifier "MA-5" and the reverse reference flag set to true. The storage controller 110 matches the manifest identifier "MA-5" with the third reverse reference entry in the reverse reference list 530A of the container index C-2 510A, thereby obtaining the data unit range R1. The storage controller 110 then decrements the reference count (in the data unit metadata 520A of the container index C-2 510A) of the data units represented by the data unit range R1. The storage controller 110 then decrements the reference count (in the data unit metadata 520A of the container index C-2 510A) of the data units represented by the data unit range R1. In another example, the storage controller 110 selects the third working entry (of C-4). Further, the storage controller 110 reads the first field in the third working entry, thereby obtaining the manifest identifier "MA-1", the reverse reference flag set to false, and the data unit range R2. The storage controller 110 then decrements the reference count (in the data unit metadata 520B of the container index C-4 510B) of the data units represented by the data unit range R2.

[0080] Figure 8 -Example housekeeping process

[0081] Figure 8 illustrates an example process 800 for housekeeping according to some embodiments. In some examples, the storage controller 110 ( Figure 1 as shown) may be used to execute the process 800. The process 800 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by one or more processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.

[0082] Block 810 may include generating, by a storage controller of a deduplication storage system, a housekeeping working map to delete backup items stored in the deduplication storage system, where the deduplication storage system stores multiple manifests to record the arrival order of multiple data units included in the backup items. Block 820 may include selecting, by the storage controller, a first working entry of the housekeeping working map, where the first working entry identifies a first container index and a first manifest.

[0083] Block 830 may include loading a first container index into a memory in response to a selection of a first work item, the first container index including a reverse reference data structure. Block 840 may include identifying, in the reverse reference data structure, a reverse reference entry indexed to a first manifest. Block 850 may include using the reverse reference entry indexed to the first manifest to determine a first set of data units included in the first manifest and indexed in the first container index. Block 860 may include decrementing a set of reference counts of the determined first set of data units in the first container index.

[0084] Figure 9 - Example machine-readable medium

[0085] Figure 9 A machine-readable medium 900 storing instructions 910-960 is shown in accordance with some embodiments. The instructions 910-960 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 900 may be a non-transitory storage medium such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium.

[0086] Instructions 910 may be executed to generate a housekeeping work map to delete backup items stored in a deduplication storage system, where the deduplication storage system stores multiple manifests to record the arrival order of multiple data units included in the backup items. Instructions 920 may be executed to select a first work item of the housekeeping work map, where the first work item identifies a first container index and a first manifest.

[0087] Instructions 930 may be executed to load the first container index into a memory in response to a selection of the first work item, the first container index including a reverse reference data structure. Instructions 940 may be executed to identify, in the reverse reference data structure, a reverse reference entry indexed to the first manifest. Instructions 950 may be executed to use the reverse reference entry indexed to the first manifest to determine a first set of data units included in the first manifest and indexed in the first container index. Instructions 960 may be executed to decrement a set of reference counts of the determined first set of data units in the first container index.

[0088] According to some embodiments of the present disclosure, a controller of a deduplication storage system may generate a housekeeping work map to perform housekeeping operations. The controller may select work entries for processing and may load a container index identified in the work entries from persistent storage into memory. The controller may read a reverse reference data structure included in the container index. Each entry of the reverse reference data structure may specify a different manifest and a data unit range. The data unit range may identify a set of data units included in the specified manifest and indexed by the container index. The controller may select reverse reference entries that specify the same manifest identified in the work entries and may extract the data unit ranges specified in the selected reverse reference entries. The controller may then decrement the reference count corresponding to the extracted data unit ranges in the container index. In this way, work entries may be processed without loading the manifest into memory. Accordingly, some embodiments may reduce the amount of I / O bandwidth used for housekeeping operations and may thereby improve the performance of the storage system.

[0089] Note that although Figures 1 to 9 various examples are shown, embodiments are not limited in this regard. For example, referring to Figure 1 , it is contemplated that the storage system 100 may include additional devices and / or components, fewer components, different components, different arrangements, etc. In another example, it is contemplated that the functionality of the storage controller 110 described above may be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0090] Data and instructions are stored in respective storage devices implemented as one or more computer-readable or machine-readable storage media. The storage media include different forms of non-transitory memory, including: semiconductor memory devices such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks such as fixed floppy disks and removable disks; other magnetic media including magnetic tape; optical media such as compact discs (CDs) or digital video discs (DVDs); or other types of storage devices.

[0091] Note that the instructions discussed above can be provided on a computer-readable or machine-readable storage medium, or alternatively, can be provided on multiple computer-readable or machine-readable storage media distributed in a large system having potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered to be part of an article (or article of manufacture). An article or article of manufacture can refer to any single manufactured component or multiple components. The one or more storage media can be located in a machine that runs the machine-readable instructions, or at a remote site from which the machine-readable instructions can be downloaded over a network for execution.

[0092] In the foregoing description, numerous details are set forth to provide an understanding of the subject matter disclosed herein. However, implementations may be practiced without some of these details. Other implementations may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A computing device, comprising: a processor; a memory; and a machine-readable storage device storing instructions that can be executed by the processor to perform the following operations: Generate a housekeeping work map to delete backup items stored in a deduplication storage system, wherein the deduplication storage system stores multiple lists to record the arrival order of multiple data units included in the backup items; Select a first work entry of the housekeeping work map, wherein the first work entry identifies a first container index and a first list; In response to the selection of the first work entry, load the first container index into the memory, the first container index including a reverse reference data structure; Identify, in the reverse reference data structure, a reverse reference entry indexed to the first list; Use the reverse reference entry indexed to the first list to determine a first set of data units included in the first list and indexed in the first container index; and Decrement a set of reference counts of the determined first set of data units in the first container index.

2. The computing device according to claim 1, comprising instructions that can be executed by the processor to perform the following operations: Determine whether a first flag included in the first work entry indicates that the combination of the first container index and the first list is to be processed using the reverse reference data structure; and In response to determining that the first flag indicates that the combination of the first container index and the first list is to be processed using the reverse reference data structure, identify, in the reverse reference data structure, the reverse reference entry indexed to the first list.

3. The computing device according to claim 2, comprising instructions that can be executed by the processor to perform the following operations: Select the second work item of the housekeeping work diagram, where The second work entry identifies a second container index and a second list; Determine whether a second flag included in the second work entry indicates that the combination of the second container index and the second list is to be processed using a second reverse reference data structure in the second container index; In response to determining that the second flag does not indicate that the combination of the second container index and the second list is to be processed using the second reverse reference data structure in the second container index: Determine a second set of data units included in the second list and indexed in the second container index based on data unit range information included in the second work entry; and and Decrement a second set of reference counts of the determined second set of data units in the second container index.

4. The computing device according to claim 1, comprising instructions that can be executed by the processor to perform the following operations: Initiate a housekeeping job to delete the backup items; Identify a set of container indices affected by the housekeeping operation, where The set of container indexes includes a third container index, and wherein the first list references the third container index; Determine whether the housekeeping work map includes an existing work entry identifying the third container index; and In response to determining that the housekeeping work diagram includes the existing work entry that identifies the third container index, add the identifier of the first list to the existing work entry.

5. The computing device according to claim 4, comprising instructions executable by the processor to perform the following operations: Determine whether the combination of the third container index and the first list is to be processed using the third reverse reference data structure in the third container index based on the item metadata representing the backup item; In response to determining that the combination of the third container index and the first list will not be processed using the third reverse reference data structure: Load the first list from persistent storage into the memory; Based on the first list loaded into the memory, determine a first data unit range that identifies a third set of data units included in the first list and indexed in the third container index; [[ID=⑥]]And Add an indication of the first data unit range to the existing work entry.

6. The computing device according to claim 5, comprising instructions executable by the processor to perform the following operations: In response to determining that the combination of the third container index and the first list will not be processed using the third reverse reference data structure, add a third flag to the existing work entry, the third flag being used to indicate that the combination of the third container index and the first list will not be processed using the third reverse reference data structure.

7. The computing device according to claim 5, comprising instructions executable by the processor to perform the following operations: In response to determining that the combination of the third container index and the first list is to be processed using the third reverse reference data structure, add a fourth flag to the existing work entry, the fourth flag being used to indicate that the combination of the third container index and the first list is to be processed using the third reverse reference data structure.

8. The computing device according to claim 4, comprising instructions executable by the processor to perform the following operations: In response to determining that the housekeeping work diagram does not include the existing work entry that identifies the third container index: Generate a new work entry that identifies the third container index in the housekeeping work diagram; and Add the identifier of the first list to the new work entry.

9. A method, comprising: Generating, by a storage controller of a deduplication storage system, a housekeeping work diagram to delete backup items stored in the deduplication storage system, wherein the deduplication storage system stores a plurality of lists to record the arrival order of a plurality of data units included in the backup items; Selecting, by the storage controller, a first work entry of the housekeeping work diagram, wherein the first work entry identifies a first container index and a first list; In response to the selection of the first work entry, load the first container index into the memory, the first container index including a reverse reference data structure; Identify a reverse reference entry indexed to the first list in the reverse reference data structure; Use the reverse reference entry indexed to the first list to determine a first set of data units included in the first list and indexed in the first container index; and Decrement a set of reference counts of the determined first set of data units in the first container index.

10. The method according to claim 9, comprising: Determine whether a first flag included in the first work entry indicates that the combination of the first container index and the first list is to be processed using the reverse reference data structure; And In response to determining that the first flag indicates that the combination of the first container index and the first list is to be processed using the reverse reference data structure, identify in the reverse reference data structure the reverse reference entry indexed to the first list.

11. The method according to claim 10, comprising: Select a second work entry of the housekeeping work map, wherein the second work entry identifies a second container index and a second list; Determine whether a second flag included in the second work entry indicates that the combination of the second container index and the second list is to be processed using a second reverse reference data structure in the second container index; In response to determining that the second flag does not indicate that the combination of the second container index and the second list is to be processed using the second reverse reference data structure in the second container index: Determine a second set of data units included in the second list and indexed in the second container index based on data unit range information included in the second work entry; and Decrement a second set of reference counts of the determined second set of data units in the second container index.

12. The method according to claim 9, comprising: Initiate a housekeeping job to delete the backup item; Identify a set of container indexes affected by the housekeeping job, wherein the set of container indexes includes a third container index, and wherein the first list references the third container index; Determine whether the housekeeping work map includes an existing work entry identifying the third container index; and In response to determining that the housekeeping work map includes the existing work entry identifying the third container index, add an identifier of the first list to the existing work entry.

13. The method according to claim 12, comprising: Determine whether the combination of the third container index and the first list is to be processed using a third reverse reference data structure in the third container index based on the item metadata representing the backup item; In response to determining that the combination of the third container index and the first list will not be processed using the third reverse reference data structure: Load the first list from persistent storage into the memory; Determine a first data unit range identifying a third set of data units included in the first list and indexed in the third container index based on the first list loaded into the memory; And Add an indication of the first data unit range to the existing work entry.

14. The method according to claim 12, comprising: In response to determining that the housekeeping work diagram does not include the existing work entry identifying the third container index: Generating a new work entry identifying the third container index in the housekeeping work diagram; and Adding the identifier of the first list to the new work entry.

15. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to perform the following operations: Generate a housekeeping work map to delete backup items stored in a deduplication storage system, where The deduplication storage system stores a plurality of lists to record the arrival order of a plurality of data units included in the backup item; Selecting a first work entry of the housekeeping work diagram, wherein the first work entry identifies a first container index and a first list; In response to the selection of the first work entry, loading the first container index into a memory, the first container index including a reverse reference data structure; Identifying, in the reverse reference data structure, a reverse reference entry indexed to the first list; Using the reverse reference entry indexed to the first list to determine a first set of data units included in the first list and indexed in the first container index; and Decrementing a set of reference counts of the determined first set of data units in the first container index.

16. The non-transitory machine-readable medium according to claim 15, comprising instructions that, when executed, cause the processor to perform the following operations: Determining whether a first flag included in the first work entry indicates that the combination of the first container index and the first list is to be processed using the reverse reference data structure; and In response to determining that the first flag indicates that the combination of the first container index and the first list is to be processed using the reverse reference data structure, identifying, in the reverse reference data structure, the reverse reference entry indexed to the first list.

17. The non-transitory machine-readable medium according to claim 16, comprising instructions that, when executed, cause the processor to perform the following operations: Select the second work item of the housekeeping work diagram, where The second work entry identifies a second container index and a second list; Determining whether a second flag included in the second work entry indicates that the combination of the second container index and the second list is to be processed using a second reverse reference data structure in the second container index; In response to determining that the second flag does not indicate that the combination of the second container index and the second list is to be processed using the second reverse reference data structure in the second container index: Determining a second set of data units included in the second list and indexed in the second container index based on data unit range information included in the second work entry; and And Decrementing a second set of reference counts of the determined second set of data units in the second container index.

18. The non-transitory machine-readable medium according to claim 15, comprising instructions that, when executed, cause the processor to perform the following operations: Initiating a housekeeping job to delete the backup item; Identify a set of container indices affected by the housekeeping operation, where, The set of container indexes includes a third container index, and wherein the first list references the third container index; Determine whether the housekeeping work map includes an existing work entry that identifies the third container index; and In response to determining that the housekeeping work map includes the existing work entry that identifies the third container index, add the identifier of the first list to the existing work entry.

19. The non-transitory machine-readable medium according to claim 18, comprising instructions that cause the processor to perform the following operations when executed: Determine whether the combination of the third container index and the first list is to be processed using a third reverse reference data structure in the third container index based on the item metadata representing the backup item; In response to determining that the combination of the third container index and the first list will not be processed using the third reverse reference data structure: Load the first list from persistent storage into the memory; Determine a first data unit range that identifies a third set of data units included in the first list and indexed in the third container index based on the first list loaded into the memory; And Add an indication of the first data unit range to the existing work entry.

20. The non-transitory machine-readable medium according to claim 18, comprising instructions that cause the processor to perform the following operations when executed: In response to determining that the housekeeping work map does not include the existing work entry that identifies the third container index: Generate a new work entry that identifies the third container index in the housekeeping work map; and Add the identifier of the first list to the new work entry.

Images