Snapshot repair in data deduplication storage system

Abstract

The invention relates to snapshot repair in a data deduplication storage system. Example implementations relate to data deduplication operations in a storage system. Examples include identifying corrupted data units included in a first snapshot; identifying a first manifest that references the corrupted data unit; a first container index that indexes the corrupted data unit based on the first manifest identification; determining a group of candidate lists based on the first container index; identifying, in the set of candidate manifests, a second manifest having a highest match score with respect to a first portion of the first manifest; and in response to determining that the identified second manifest having the highest match score does not reference the corrupted data unit, replacing the reference to the corrupted data unit in the first manifest with a reference to an uncorrupted data unit referenced in the identified second manifest.

Description

Background Art

[0001] Data reduction techniques may be applied to reduce the amount of data stored in a storage system. Example data reduction techniques include data deduplication. Data deduplication identifies duplicate data units and seeks to reduce or eliminate the number of instances of the duplicate data units stored in a storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Some embodiments are described with respect to the following figures.

[0003] Figure 1 is a schematic diagram of an example storage system according to some implementations.

[0004] FIG. 2A to FIG. 2C is an illustration of an example data structure according to some implementations.

[0005] Figure 3 is an illustration of an example process according to some implementations.

[0006] FIG. 4A to FIG. 4M is an illustration of example operations according to some implementations.

[0007] Figure 5 is an illustration of an example process according to some implementations.

[0008] Figure 6 is a schematic diagram of an example computing device in accordance with some implementations.

[0009] Figure 7 is an illustration of an example process according to some implementations.

[0010] Figure 8 is a diagram of an example machine-readable medium having instructions stored thereon, according to some implementations.

[0011] Throughout the drawings, the same reference numerals refer to similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the size of some parts may be exaggerated to more clearly illustrate the examples shown. In addition, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0012] In the present disclosure, unless the context clearly indicates otherwise, the use of the terms "a", "an" or "the" is intended to include the plural forms as well. Similarly, when used in the present disclosure, the terms "includes / including / comprises / comprising" or "have / having" specify the presence of the elements, but do not exclude the presence or addition of other elements.

[0013] In some examples, a storage system may back up a collection of data (referred to herein as a "stream" or "data stream" of data) in a deduplicated form, thereby reducing the amount of storage space required to store the data stream. The storage system may create a "backup item" to represent the data stream in a deduplicated form. The storage system may perform a deduplication process including breaking the data stream into discrete data units (or "chunks") and determining "fingerprints" of these incoming data units (as described below). Further, the storage system may compare the fingerprints of the incoming data units with the fingerprints of the stored data units, and may thereby determine which incoming data units are duplicates of previously stored data units (e.g., when the comparison indicates matching fingerprints). In the event that the data units are duplicates, the storage system may store a reference to the previously stored data unit instead of storing the duplicate incoming data unit. The process for receiving and deduplicating an incoming data stream may be referred to herein as a "data ingestion" process of the storage system.

[0014] As used herein, "fingerprint" refers to a value obtained by applying a function to the contents of a data unit (where "content" may include all or a subset of the contents of a data unit). Examples of functions that may be applied include hash functions that produce a hash value based on the contents of an incoming data unit. Examples of hash functions include cryptographic hash functions, such as a Secure Hash Algorithm 2 (SHA-2) hash function (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions may be employed.

[0015] A "storage system" may include a storage device or an array of storage devices. A storage system may also include (multiple) storage controllers that manage access to (multiple) storage devices. A "data unit" may refer to any data portion that can be individually identified in a storage system. In some cases, a data unit may refer to a block, a collection of blocks, or any other data portion. In some examples, a storage system may store data units in a persistent storage device. A persistent storage device may be implemented using one or more (multiple) persistent (e.g., non-volatile) storage devices (such as (multiple) disk-based storage devices (e.g., (multiple) hard disk drives (HDDs)), (multiple) solid-state devices (SSDs) (such as (multiple) flash storage devices), etc. or a combination thereof. A "controller" may refer to a hardware processing circuit, which may include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuits. Alternatively, a "controller" may refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuit.

[0016] In some examples, a deduplication storage system may use metadata to process an inbound data stream (e.g., a backup project). For example, such metadata may include a data recipe (also referred to herein as a "manifest") that specifies the order in which specific data units of each backup project are received. Further, such metadata may include project metadata to represent each received backup project in a deduplicated form. The project metadata may include an identifier for a set of manifests and may indicate the order of the set of manifests. Processing each backup project may be referred to herein as a "backup process." Subsequently, in response to a read request, the deduplication system may use the project metadata and the set of manifests to determine the order in which the data units are received, so that the original data stream of the backup project can be recreated. Accordingly, the set of manifests may be a representation of the original backup project. The manifest may include a series of records, each record representing a specific set (or multiple) of data units. The records in the manifest may include one or more fields identifying container indexes that index the data units (e.g., including storage information of the data units). For example, the container index may include one or more fields that specify location information (e.g., container, offset, etc.) of the stored data units, compression and / or encryption characteristics of the stored data units, and the like. Further, the container index may include reference counts indicating the number of manifests that reference each data unit.

[0017] In some examples, when a data unit (e.g., in a data stream) is received, it can be matched with one or more container indexes to determine whether the same chunk is already stored in a container of a deduplication storage system. For example, the deduplication storage system can compare the fingerprint of the received data unit with the fingerprint in one or more container indexes. If a matching fingerprint is not found in the searched container index (s), the received data unit can be added to the container, and the entry of the received data unit can be added to the container index corresponding to the container. However, if a matching fingerprint is found in the searched container index, it can be determined that the same data unit as the received data unit is already stored in the container. In response to the determination, the reference count of the corresponding entry is incremented, and the received data unit is not stored in the container (because it already exists in one of the containers), thereby avoiding the storage of duplicate data units in the deduplication storage system. As used herein, the term "matching operation" can refer to an operation for comparing the fingerprint of a set of multiple data units (e.g., from a particular backup data stream) with the fingerprint stored in the container index.

[0018] In some examples, a deduplication storage system may process an inbound data stream to store deduplication copies (also referred to herein as "snapshots") of all data blocks in a source data set (also referred to herein as a "source project") at a specific point in time. Further, in some examples, at least some of the data blocks in the source project may change over time. For example, a sales database (i.e., a source project) may be copied in a snapshot at a first point in time, and the sales database may be subsequently updated to include new records of additional sales transactions. Accordingly, a deduplication storage system may generate and store a sequence of snapshots to capture the changing state of the source project at different points in time. In some examples, some or all of the data units in the source project may be lost or become unavailable (e.g., due to malware attacks, system failures, etc.). In such examples, the stored snapshots may be used to restore at least some of the lost data of the source project. For example, if the sales database is lost due to a system failure, the most recent snapshot may be copied to regenerate the sales database that existed at the time of the snapshot. However, such a copy of the most recent snapshot will not restore changes to the sales database that occurred after the most recent snapshot was created.

[0019] In some examples, data units included in a snapshot may be damaged. As used herein, the term "damaged" may refer to data units that do not include correct data content. For example, a data unit in a source project may be damaged by a malware attack, and the damaged data unit may be copied to each subsequent snapshot of the source project. In this way, the (multiple) most recent snapshots that include the damaged data unit may not be available to repair the source project. Therefore, it may be necessary to use an older snapshot that does not include the damaged data unit to perform repairs to the source project. Further, because the older snapshot is less similar to the source project than the most recent snapshot (for example, because changes to the source project accumulate over time), the older snapshot must be used when restoring the source project, which is likely to result in greater data loss.

[0020] According to some embodiments of the present disclosure, a controller of a deduplication storage system can repair a snapshot including a damaged data unit. The controller can identify the damaged data unit included in the snapshot of the source project. The controller can identify a first list that references the snapshot portion including the damaged data unit. The controller can then select a comparison window (e.g., a set of data units including the damaged data unit) from the first list, and can compare the comparison window with a set of candidate lists associated with a previous snapshot of the source project. The controller can assign a matching score to each candidate list based on the similarity of each candidate list to the comparison window. If the candidate list with the highest matching score also references the damaged data unit, the controller can select a new comparison window from the candidate list, and can then compare the remaining candidate lists with the new comparison window. When it is determined that the candidate list with the highest score does not reference the damaged data unit, the controller can identify the undamaged data unit from the candidate list, and can determine that the undamaged data unit should be used in the snapshot instead of the damaged data unit. Further, the controller can replace the reference to the damaged data unit with a reference to the undamaged data unit (in the first list). In this way, the controller can repair the affected list with reduced (or no) data loss, and thus can improve the performance of the storage system. Figures 1 to 8 Various aspects of the disclosed repair process are discussed further.

[0021] Figure 1 - Sample storage system

[0022] Figure 1An example of a storage system 100 including a storage controller 110, a memory 115, and a persistent storage device 140 according to some embodiments is shown. The persistent storage device 140 may include one or more non-transitory storage media, such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc., or a combination thereof. The memory 115 may be implemented with a semiconductor memory such as a random access memory (RAM). In some examples, the storage controller 110 may be implemented via hardware (e.g., electronic circuits) or a combination of hardware and programs (e.g., including at least one processor and instructions executable by the at least one processor and stored on at least one machine-readable storage medium).

[0023] like Figure 1 As shown, memory 115 and persistent storage 140 may store various data structures including at least project metadata 130, manifest 150, container index 160, data container 170, and alias list 180. In some examples, copies of project metadata 130, manifest 150, container index 160, data container 170, and alias list 180 may be transferred between memory 115 and persistent storage 140 (e.g., via read and write input / output (I / O) operations).

[0024] In some embodiments, storage system 100 may perform a data ingestion operation to deduplicate received data. For example, storage controller 110 may receive an inbound data stream comprising a plurality of data units, and may store at least one copy of each data unit in data container 170 (e.g., by appending the data unit to the end of data container 170). In some examples, each data container 170 may be divided into entities, wherein each entity comprises a plurality of stored data units. Further, in some examples, the inbound stream may be deduplicated and stored as a backup item.

[0025] In one or more embodiments, the storage controller 110 may generate a fingerprint for each received data unit. For example, the fingerprint may include a complete or partial hash value based on the data unit. In order to determine whether the incoming data unit is a duplicate of a stored data unit, the storage controller 110 may perform a matching operation to compare the fingerprint generated for the incoming data unit with the fingerprint in the at least one container index 160. If a match is identified, the storage controller 110 may determine that the storage system 100 has stored a duplicate of the incoming data unit. The storage controller 110 may then store a reference to the previous data unit instead of storing the duplicate incoming data unit.

[0026] In some implementations, the storage controller 110 may generate item metadata 130 to represent each backup item in a deduplicated form. In some examples, a set of backup items (represented by a set of item metadata 130) may be a sequence of snapshots that captures the changing data content of a source item (e.g., a storage volume) at multiple points in time. Each item metadata 130 may include an identifier of a set of manifests 150 and may indicate an order of the set of manifests 150. The manifests 150 record the order in which the data units are received.

[0027] In some embodiments, the manifest 150 may include a pointer or other information indicating a container index 160 that indexes each data unit. In some embodiments, the container index 160 may include a fingerprint (e.g., a hash) of the stored data unit for use in a matching process of a deduplication process. Further, the container index 160 may indicate the location where the data unit is stored. For example, the container index 160 may include information specifying that the data unit is stored at a specific offset in the entity and that the entity is stored at a specific offset in the data container 170. The container index 160 may also include a reference count indicating the number of manifests 150 that reference each data unit.

[0028] In some embodiments, the storage controller 110 may receive a read request to access the stored data, and in response may access the project metadata 130 and the manifest 150 to determine the sequence of data units that constitute the original data. The storage controller 110 may then use the pointer data included in the manifest 150 to identify the container index 160 that indexes the data unit. Further, the storage controller 110 may use the information included in the identified container index 160 (as well as the information included in the manifest 150) to determine the location (e.g., data container 170, entity, offset, etc.) where the data unit is stored, and may then read the data unit from the determined location.

[0029] In some embodiments, the storage controller 110 may identify one or more corrupted data units included in the first snapshot of the source project, and may identify a manifest 150 that references the corrupted data unit(s). For example, the storage controller 110 may receive (e.g., from a user or program) a message or command that identifies a data range in the snapshot that includes the corrupted data unit (e.g., based on a scan or analysis of the snapshot). In some examples, the location of the corrupted data unit may be specified as an offset and size in a given backup project (e.g., project metadata 130). In such an example, the storage controller 110 may determine a specific manifest 150 that represents the offset and size in the project metadata 130.

[0030] In some embodiments, the storage controller 110 may select a set of data unit references (also referred to herein as a "comparison window") in the list 150 that includes a reference to a corrupted data unit. For example, the storage controller 110 may select a comparison window in the list 150 that includes a subset of data unit references, wherein the reference to the corrupted data unit is located at the center of the comparison window (also referred to herein as a "center position"). Figure 4B Example implementations of comparison windows are described.

[0031] In some embodiments, the storage controller 110 may identify a set of candidate lists 150 that reference data units included in a previous snapshot of the source project (e.g., multiple snapshots generated before the snapshot that includes the identified corrupted data unit). The storage controller 110 may compare a comparison window with each candidate list 150, and may assign a match score to each candidate list based on the similarity of each candidate list 150 to the comparison window. As used herein, the term "match score" may refer to a numerical value that measures the similarity between a data unit referenced in a comparison window and a set of data units referenced in a candidate list 150. For example, a match score may indicate how many data units are referenced in both the comparison window and a portion of the candidate list 150. Reference is made below to the description of the match score. Figures 4D to 4G Describes an example calculation of the match score.

[0032] In some embodiments, the storage controller 110 may identify the candidate list 150 with the highest match score (e.g., most similar to the comparison window). Further, if the identified candidate list 150 also references a corrupted data unit, the storage controller 110 may select a new comparison window from the identified candidate list 150, and then the remaining candidate lists 150 may be compared with the new comparison window. If necessary, the storage controller 110 may repeat the process (e.g., determining that the candidate list 150 with the highest match score references the corrupted data unit, and then selecting a new comparison window) in multiple iterations until it is determined that the candidate list 150 with the highest match score does not reference the corrupted data unit. Further, when it is determined that the candidate list 150 with the highest match score does not reference the corrupted data unit, the storage controller 110 may select the uncorrupted data unit referenced in the candidate list 150. In some embodiments, the uncorrupted data unit may be determined to be the correct data unit that should exist in the snapshot, rather than the corrupted data unit. Therefore, the storage controller 110 can use the uncorrupted data unit to repair the manifest 150 that references the corrupted data unit (e.g., by replacing the reference to the corrupted data unit with the reference to the uncorrupted data unit). In this way, the storage controller 110 can repair the manifest 150 with reduced (or no) data loss, and thus can improve the performance of the storage system 100. Figure 3 and FIG. 4A to FIG. 4M Describes an example procedure for repairing a manifest included in a snapshot.

[0033] In some embodiments, the storage controller 110 may create an entry in the alias list 180 to store information indicating that an uncorrupted data unit has been determined to be a replacement for a corrupted data unit. An uncorrupted data unit may be referred to herein as an "alias" for a corrupted data unit that it may replace (e.g., in a reference in the manifest 150). The storage controller 110 may then compare the received data unit to the alias list 180. If the received data unit matches a corrupted data unit listed in an entry of the alias list 180, the storage controller 110 may replace the received data unit in the manifest 150 with the uncorrupted data unit listed in the entry (i.e., the alias of the corrupted data unit). hereinafter referenced Figure 5 An example process for using alias list 180 is described.

[0034] FIG. 2A to FIG. 2C - Example data structure

[0035] Figure 2A 2 shows an example data structure 200 for use in deduplication according to some embodiments. As shown, data structure 200 may include project metadata 202, manifest 203, container index 220, and data container 250. In some examples, project metadata 202, manifest 203, container index 220, and data container 250 may generally correspond to project metadata 130, manifest 150, container index 160, and data container 170, respectively (e.g., Figure 1 In some examples, data structure 200 may be stored by storage controller 110 (eg, Figure 1 as shown) to generate and / or manage.

[0036] In some implementations, the item metadata 202 may include a plurality of manifest identifiers 205. Each manifest identifier 205 may identify a different manifest 203. In some implementations, the manifest identifiers 205 may be arranged in a stream order (i.e., based on the order in which the data units represented by the identified manifest 203 are received). Further, the item metadata 202 may include a container list 204 associated with each manifest identifier 205. In some implementations, the container list 204 may include an identifier of a set of container indexes 220 that index the data units included in the associated manifest 203 (i.e., the manifest 203 identified by the associated manifest identifier 205).

[0037] Although for simplification Figure 2AIn the illustration of FIG, only one of each data structure is shown, but the data structure 200 may include multiple instances of the project metadata 202, each instance including or pointing to one or more manifests 203. In such an example, the data structure 200 may include multiple manifests 203. The manifest 203 may reference multiple container indexes 220, each container index corresponding to one of the multiple data containers 250. Each container index 220 may include one or more data unit records 230 and one or more entity records 240.

[0038] like Figure 2A As shown, in some examples, each manifest 203 may include one or more manifest records 210. Each manifest record 210 may include various fields, such as offset, length, container index, and unit address. In some embodiments, each container index 220 may include any number of data unit records 230 and entity records 240. Each data unit record 230 may include various fields, such as fingerprint (e.g., hash of data unit), unit address, entity identifier, unit offset (i.e., offset of data unit within entity), reference count value, and unit length. In some examples, the reference count value may indicate the number of manifest records 210 that reference the data unit record 230. Further, each entity record 240 may include various fields, such as entity identifier, entity offset (i.e., offset of entity within container), storage length (i.e., length of data unit within entity), decompressed length, checksum value, and compression / encryption information (e.g., compression type, encryption type, etc.). In some embodiments, each data container 250 may include any number of entities 260, and each entity 260 may include any number of stored data units.

[0039] In some implementations, a unit address (included in manifest record 210 and data unit record 230) may be an identifier that deterministically identifies a particular data unit within a given container index 220. In some examples, a unit address may be a numerical value (referred to as an "arrival number") that indicates (e.g., when an inbound data stream is received and deduplicated) the order of arrival (also referred to as the "ingestion order") of data units indexed in a given container index 220. For example, a first data unit to be indexed in a container index 220 (e.g., by creating a new data unit record 230 for the first data unit) may be assigned an arrival number of "1", a second data unit may be assigned an arrival number of "2", a third data unit may be assigned an arrival number of "3", and so on. However, other implementations are possible.

[0040] In some embodiments, the manifest record 210 may use a run-length reference format to represent a range of consecutive data units (e.g., a portion of a data stream) indexed within a single container index 220. The run-length reference may be recorded in a unit address field and a length field of the manifest record 210. For example, the unit address field may indicate an arrival number of a first data unit in the represented data unit range, and the length field may indicate the number N (where "N" is an integer) of data units in the data unit range that are subsequent to the data unit specified by the arrival number in the unit address field. The data units in the data unit range may have consecutive arrival numbers (e.g., because they are consecutive in the ingested data stream). Thus, the data unit range may be represented by the arrival number of the first data unit in the data unit range (e.g., specified in the unit address field of the manifest record 210) and the number N of additional data units in the data unit range (e.g., specified in the length field of the manifest record 210). Additional data units in the data unit range that follow the first data unit can be deterministically obtained by calculating the N arrival numbers that are sequentially subsequent to the designated arrival number of the first data unit, where the N arrival numbers identify additional data units in the data unit range. In such an example, the manifest record 210 can include an arrival number "X" in the unit address field and a number N in the length field to indicate a data unit range that includes the data unit specified by the arrival number X and the data unit specified by the arrival number X+i, where i=0 to i=N (inclusive) (where "i" is an integer). In this manner, the manifest record 210 can be used to identify all data units in the data unit range.

[0041] In one or more embodiments, the data structure 200 can be used to obtain stored deduplicated data. For example, a read request can specify the offset and length of the data in a given file. These request parameters can be matched with the offset and length fields of a specific inventory record 210. The container index and unit address of the specific inventory record 210 can then be matched with the specific data unit record 230 included in the container index 220. Further, the entity identifier of the specific data unit record 230 can be matched with the entity identifier of the specific entity record 240. In addition, one or more other fields of the specific entity record 240 (e.g., entity offset, storage length, checksum, etc.) can be used to identify the container 250 and the entity 260, and then the data unit can be read from the identified container 250 and the entity 260.

[0042] In some implementations, each container index 220 may include an inventory list 222. An inventory list 222 may be a data structure for storing a set of entries, where each entry stores information about a different inventory 203 (or inventory record 210) indexed by a container index 220 (including an inventory list 222). For example, each time a container index 220 is generated or updated to include information about a particular inventory record 210, the inventory list 222 in the container index 220 is updated to store an identifier of the inventory record 210. Further, the entries of the inventory list 222 may be arranged in the respective order in which they entered the inventory list 222 (also referred to herein as the "arrival order" of the entries). In some examples, when a container index 220 is no longer associated with an inventory record 210, the identifier of the inventory record 210 is removed from the inventory list 222.

[0043] Reference now Figure 2B , shows an example implementation of the inventory list 222. Figure 2A As shown, in some embodiments, each entry of manifest list 222 may store only manifest identifier 205. However, other examples are possible. For example, each entry of manifest list 222 may instead include manifest identifier 205 and at least one data unit range (e.g., a group of one or more data units included in manifest 203 and indexed by container index 220).

[0044] Reference now Figure 2C , shows an example implementation of the alias data structure 270. The alias data structure 270 may generally correspond to the alias list 180 (eg, Figure 1 ) as shown in the example implementation. Figure 2C As shown, the alias data structure 270 may include a plurality of entries, wherein each entry stores an identifier (e.g., a fingerprint) of a corrupted data unit and its alias (i.e., an uncorrupted data unit that has been determined to be an alternative to the corrupted data unit). In some implementations, an entry may be added to the alias data structure 270 when a particular uncorrupted data unit is determined to be an alias of a particular corrupted data unit.

[0045] Figure 3 - Example procedure for repairing a snapshot

[0046] Figure 3 An example process 300 for repairing a snapshot in a deduplication storage system according to some embodiments is shown. In some examples, a storage controller 110 (eg, Figure 1300). Process 300 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 300 may be executed by a single processing thread. In other embodiments, process 300 may be executed in parallel by multiple processing threads (e.g., using a work map and executing multiple housekeeping jobs simultaneously).

[0047] Block 310 may include identifying corrupted data units included in a first snapshot of the backup project. Block 320 may include identifying a first manifest referencing the corrupted data units. In some implementations, the manifest may reference a total number of T (where "T" is an integer) of data units, and the order in which the T data units were received may be recorded in the manifest. For example, with reference to Figure 1 and Figure 4A , storage controller 110 receives an indication (e.g., a user command, a software message, etc.) identifying a data range (e.g., an offset and length in a backup project) in a snapshot that includes a corrupt data unit 412 (or multiple corrupt data units). In some examples, the presence of the corrupt data unit 412 may be detected by malware scanning, integrity testing, etc. Storage controller 110 reads the project metadata 130 representing the snapshot, identifies a manifest 410 that references the corrupt data unit 412 (i.e., data unit "X"), and loads the manifest 410 into memory 115. For simplicity, Figure 4A Manifest 410 is shown as including a total of seven data units, T. However, other sizes and arrangements of manifest 410 are possible.

[0048] Block 330 may include identifying a comparison window in the first manifest. In some embodiments, the comparison window may have a width W (where "W" is an integer) indicating the number of data units referenced in the comparison window, wherein the width W is less than the total number "T" of data units referenced in the manifest. Further, in some embodiments, the width W may be an odd integer (i.e., having odd parity) such that a particular data unit is referenced at a center position in the comparison window. For example, reference Figure 1 and Figure 4B , the storage controller selects a comparison window 414 having a width W equal to five (i.e., referencing five different data units) in the list 410. In some embodiments, the corrupted data unit 412 is referenced at the center of the comparison window 414. However, other sizes and / or arrangements of the comparison window 414 are possible.

[0049] Block 340 may include identifying a first container index for indexing the corrupted data unit based on the first manifest. Block 350 may include identifying a set of candidate manifests based on the manifest list included in the first container index. For example, referring to Figures 1 to 2A and Figure 4C , the storage controller 110 reads the manifest 150 to identify the container index 160 that indexes the corrupted data unit, and then loads the container index 160 into the memory 115. Further, the storage controller 110 reads the manifest list 222 (stored in the container index 160) to identify the manifest set 416 (e.g., candidate manifests 420, 430, 440, 450) indexed by the container index 160. In some embodiments, the set 416 is arranged in arrival order (e.g., in the order of entry into the manifest list 222). The storage controller 110 loads each candidate manifest in the set 416 into the memory 115 (e.g., one at a time or as a group). In some examples, each candidate manifest in the set 416 may reference data units in different snapshots of a particular source project. In such an example, each candidate manifest may record the order of data units in a particular portion of the source project at different points in time (i.e., corresponding to each snapshot).

[0050] Block 360 may include determining a match score based on a sliding comparison of the set of candidate lists with a comparison window. Block 370 may include selecting the candidate list with the highest match score. As used herein, "sliding comparison" may refer to an operation that includes sliding a selection window (also referred to as a "scanning window") across the candidate list to select different subsets of data units referenced in the candidate list, and comparing each subset of data units with the data units referenced in the current comparison window. In some embodiments, the width of the scanning window used in the sliding comparison may be equal to the width W of the comparison window. For example, with reference to Figure 4D , the controller selects (in set 416) the candidate list 420 that is closest to the list 410 (identified at box 320). The controller initiates a sliding comparison of the candidate list 420 by placing the scanning window 422 in the first position, thereby selecting the leftmost W data units referenced in the candidate list 420 (e.g., the earliest W data units in arrival order), where W is the width of the comparison window 414. The controller then determines a similarity value between the comparison window 414 and the scanning window 422 in the first position. In some embodiments, the similarity value can be calculated as the total number of non-central data units (i.e., data units that are not referenced in the center position of the window) that are referenced in both the comparison window 414 and the current scanning window 422. For example, if Figure 4DAs shown, two non-central data units 423 (ie, data units "32" and "42") are referenced in both comparison window 414 and scan window 422. Therefore, the similarity value of scan window 422 in the first position is equal to two.

[0051] Reference now Figure 4E , the controller continues the sliding comparison of the candidate list 420 by sliding the scanning window 422 to the right by one data unit reference to reach the second position (ie, selecting data units "32", "42", "X", "19" and "76"). Figure 4E As shown, the controller then identifies three non-central data units 425 (i.e., data units "32", "42", and "76") that are referenced in both the comparison window 414 and the second position scanning window 422. Therefore, the similarity value of the second position scanning window 422 is equal to three.

[0052] Reference now Figure 4F , the controller completes the sliding comparison of the candidate list 420 by sliding the scanning window 422 to the right by one data unit reference to reach the third position (i.e., selecting data units "42", "X", "19", "76" and "81"). Figure 4F As shown, the controller then identifies two non-central data units 427 (ie, data units "42" and "76") that are referenced in both the comparison window 414 and the third-position scan window 422. Therefore, the similarity value of the third-position scan window 422 is equal to two.

[0053] Reference now Figure 4G , the controller determines that the highest similarity score (i.e., three points) in the sliding comparison of the candidate list 420 is at the second position (e.g., Figure 4E ). Therefore, the controller identifies the second position of the scanning window 422 as the matching window 424 of the candidate list 420. As used herein, a "matching window" may refer to a subset of data unit references in the candidate list that have the highest similarity with respect to the data unit references in the current comparison window (i.e., the comparison window being used for the sliding comparison). Further, as Figure 4G As shown, the controller sets the matching score of candidate list 420 to three (“S=3”), corresponding to the similarity score of matching window 424 .

[0054] Reference now Figure 4H , the controller performs a sliding comparison of set 416 (ie, lists 420, 430, 440, and 450) with comparison window 414 and determines the matching scores of the candidate lists. Further, the controller selects the candidate list 420 in set 416 with the best matching score (ie, three).

[0055] Reference again Figure 3 , decision block 375 may include determining whether the selected manifest references a corrupt data unit (identified at block 310). If it is determined that the selected manifest references a corrupt data unit ("yes"), process 300 may continue at block 380, including removing the selected manifest from the set of candidate manifests. Block 385 may include identifying a new comparison window in the selected manifest. After block 385, process 300 may return to block 360 (i.e., determining a new match score based on a sliding comparison of the remaining candidate manifests with the new comparison window). In some examples, blocks 360, 370, 375, 380, and 385 may be repeated in one or more loops until a negative determination is reached at decision block 375 (i.e., when it is determined that the selected candidate manifest does not reference a corrupt data unit). For example, with reference to Figure 4H , the controller determines that in the candidate list 420 (selected at block 370), the center position of the matching window 424 references the corrupted data unit 412 (which is also referenced in the list 410). In response to this determination, Fig. 4I As shown, the controller removes the candidate list 420 from the set 416 to obtain the reduced set 417 , and selects the matching window 424 in the candidate list 420 as the second comparison window 428 .

[0056] Reference now Figure 4J , the controller performs a sliding comparison of the reduced set of candidate lists 417 (i.e., lists 430, 440, 450) with the second comparison window 428, and determines the match scores of these candidate lists. The controller then determines that candidate list 430 has the best match score (i.e., three), corresponding to the similarity score of the second matching window 434 in candidate list 430. Further, the controller determines that the center position of the second matching window 434 references the corrupted data unit 412. In response to this determination, as Figure 4K As shown, the controller removes the candidate list 430 from the reduced set 417 to obtain the second reduced set 418 , and selects the second matching window 434 in the candidate list 430 as the third comparison window 438 .

[0057] Reference again Figure 3 If, at decision block 375, it is determined that the selected manifest does not reference a corrupt data unit ("No"), process 300 may continue at block 390, including identifying an uncorrupted data unit referenced in the selected manifest. Block 395 may include replacing the reference to the corrupt data unit in the first manifest with a reference to the identified uncorrupted data unit. After block 395, process 300 may be complete. For example, referring to Figure 4L, the controller performs a sliding comparison of the second reduced set 418 (i.e., lists 440, 450) with the third comparison window 438, and determines the match scores of these candidate lists. The controller then determines that candidate list 440 has the best match score (i.e., three), corresponding to the similarity score of the third matching window 444 in candidate list 440. Further, the controller determines that the center position of the third matching window 444 does not reference the corrupted data unit 412, but rather references the uncorrupted data unit 445 (i.e., data unit "04"). In response to this determination, as Figure 4M As shown, the controller uses the uncorrupted data unit 445 in the list 410, 420, 430 (i.e., previously in Figure 3 The controller may then perform repair 460 (i.e., by replacing references to corrupt data unit 412 with references to uncorrupted data unit 445) in each manifest in the list (i.e., manifests determined to include corrupt data units at decision block 375 as shown). In this manner, the controller may repair the affected manifests with reduced (or no) data loss, and may thereby improve performance of the system including the stored data.

[0058] In some implementations, the controller can select a name in the alias list (e.g. Figure 1 4. The controller may create an entry in the alias list 180 shown in FIG. 1 to indicate that the uncorrupted data unit 445 has been determined to be a replacement (i.e., an alias) for the corrupted data unit 412. Subsequently, if a received data unit matches the corrupted data unit 412, the controller may use the alias list to replace the received data unit with the uncorrupted data unit 445. Figure 5 Describes an example procedure for using an alias list.

[0059] In some implementations, a match score is determined based on a sliding comparison (e.g., Figure 3 360) can be performed using comparison windows of different sizes. For example, the controller may perform an initial set of sliding comparisons using a first width of the comparison window (e.g., W=3), and may not be able to determine the best matching window (in the candidate list) based on the matching scores (e.g., if all matching scores are equal, if all matching scores are below a minimum threshold for useful matching scores, etc.). In this example, the controller may initiate a second set of sliding comparisons using a larger width of the comparison window (e.g., W=5), and may again attempt to determine the best matching window based on the matching scores. The process may be repeated (with increasing widths of the comparison window) until valid results are obtained from the sliding comparisons (i.e., the best matching window is determined in the candidate list).

[0060] Note that the implementation is not limited to FIG. 4A to FIG. 4MFor example, it is contemplated that any or all of comparison windows 414 may not be located at the center of list 410. Similarly, match window 424 may not be located at the center of candidate list 420, and second match window 434 may not be located at the center of candidate list 430. Further, although Figure 4H An example is shown where the selected candidate manifest 420 having the best match score is directly adjacent to the first manifest 410 (including the comparison window 414), but it is contemplated that in other examples, the selected candidate manifest may not be adjacent to the manifest including the comparison window. For example, if candidate manifest 420 instead has a lower match score than candidate manifest 430, then candidate manifest 430 may be selected as having the highest match score in set 416. In this example, candidate manifest 420 may not reference the same data unit stream (e.g., in a particular source project) referenced by other candidate manifests in set 416.

[0061] Figure 5 - Example procedures for using alias lists

[0062] Figure 5 An example process 500 for using an alias list in a deduplication storage system according to some embodiments is shown. In some examples, a storage controller 110 (eg, Figure 1 Process 500 may be performed by a computer program that is configured to execute the process 500. Process 500 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 500 may be executed by a single processing thread. In other embodiments, process 500 may be executed in parallel by multiple processing threads (e.g., using a work map and executing multiple housekeeping jobs simultaneously).

[0063] Block 510 may include identifying a manifest to be repaired using an uncorrupted data unit. Block 520 may include replacing references to the corrupted data unit in the identified manifest with references to the uncorrupted data unit. For example, reference FIG. 4A to FIG. 4M The controller identifies a corrupted data unit 412 in the snapshot, identifies a manifest 410 that references the corrupted data unit 412, and selects a comparison window 414 in the manifest 410. The controller identifies a container index that indexes the corrupted data unit 412, and identifies a candidate manifest set 416 indexed by the container index (e.g., using Figure 2B420). Further, the controller determines a match score based on a sliding comparison of the candidate list set 416 with the comparison window 414, selects a match window 424 with the highest match score in the candidate list 420, and determines whether the match window 424 includes a reference to the corrupted data unit 412. If so, the controller performs one or more iterations of selecting a comparison window, performing a sliding comparison of the remaining candidate lists, and selecting a match window from the remaining candidate lists. Upon determining that the selected match window (e.g., the third match window 444 in the candidate list 440) does not include a reference to the corrupted data unit 412, the controller replaces the reference to the corrupted data unit 412 in the affected list with a reference to an uncorrupted data unit 445 in the candidate subset 444.

[0064] Block 530 may include incrementing the reference count value of the uncorrupted data unit in the container index. Block 540 may include deleting the corrupted data unit and associated metadata. For example, referring to Figure 1 , the storage controller 110 increments the reference count representing the non-corrupt data unit (in the container index 160). Further, the controller deletes the corrupted data unit in the data container 170 and deletes the metadata associated with the corrupted data unit (eg, in the container index 160).

[0065] Block 550 may include storing alias information indicating that the undamaged data unit is a repair replacement for the damaged data unit. Figure 1 , the storage controller 110 adds a specific entry to the alias list 180, wherein the specific entry indicates that the damaged data unit can be repaired or replaced by the undamaged data unit.

[0066] Block 560 may include comparing the received data unit to the alias information to detect instances of corrupted data units. Block 570 may include recording a reference to an uncorrupted data unit based on the alias information when an instance of a corrupted data unit is detected. For example, referring to Figure 1, the storage controller 110 receives a stream of data units to be stored in the storage system 100, and compares the received data units to entries of the alias list 180 (e.g., by comparing the fingerprint of the received data unit to the fingerprint stored in the alias list 180, identifying the damaged data unit that has been determined to have a repair replacement). Upon determining that the received data unit matches the damaged data unit listed in a particular entry of the alias list 180, the storage controller 110 does not record a reference to the damaged data unit in the list 150 that records the order of the received stream. Instead, the storage controller 110 reads the particular entry to identify the undamaged data unit, and then records the identified undamaged data unit in the list 150 (i.e., in place of the damaged data unit). Therefore, the damaged data unit is not stored as part of the deduplication backup item (representing the received stream). In this way, some embodiments can reduce the amount of damaged data stored in the storage system 100. Further, some embodiments can reduce the data loss and performance impact associated with recovering from each instance of the damaged data unit.

[0067] Figure 6 - Sample computing device

[0068] Figure 6 1 shows a schematic diagram of an example computing device 600. In some examples, computing device 600 may generally correspond to storage system 100 (eg, Figure 1 As shown, computing device 600 may include hardware processor 602, memory 604, and machine-readable storage device 605 including instructions 610-660. Machine-readable storage device 605 may be a non-transitory medium. Instructions 610-660 may be executed by hardware processor 602 or by a processing engine included in hardware processor 602.

[0069] Instructions 610 may be executed to identify corrupted data units included in a first snapshot stored in a deduplication storage system. Instructions 620 may be executed to identify a first manifest referencing the corrupted data units, wherein the first manifest records an ingestion order of the data units of the first snapshot into the deduplication storage system. For example, referring to Figure 1 and Figure 4A , storage controller 110 receives an indication (e.g., a user command, a software message, etc.) identifying a corrupt data unit 412 (or multiple corrupt data units). In some examples, the corrupt data unit 412 may be detected by malware scanning, integrity testing, validity testing, etc. Storage controller 110 reads project metadata 130 representing the snapshot, identifies a manifest 410 including the corrupt data unit 412, and loads the manifest 410 into memory 115.

[0070] Instructions 630 may be executed to identify a first container index for indexing corrupted data units based on the first list. Instructions 640 may be executed to determine a set of candidate lists based on the first container index. For example, referring to Figures 1 to 2A and Figure 4C , the storage controller 110 reads the manifest 150 to identify the container index 160 that indexes the corrupted data unit, and then loads the container index 160 into the memory 115. Further, the storage controller 110 reads the manifest list 222 (stored in the container index 160) to identify a set 416 of candidate manifests indexed by the container index 160, and loads each candidate manifest in the set 416 into the memory 115 (e.g., one at a time or as a group).

[0071] Instructions 650 may be executed to identify, in the set of candidate lists, a second list having a highest match score with respect to the first portion of the first list. Figure 1 and Figure 4B , selecting the comparison window 414 in the list 410 in the storage controller 110 .

[0072] Instructions 660 may be executed to, in response to determining that the identified second manifest with the highest match score does not reference the corrupted data unit, replace the reference to the corrupted data unit in the first manifest with a reference to the uncorrupted data unit referenced in the identified second manifest. FIG. 4D to FIG. 4M , the controller determines a match score based on a sliding comparison of the candidate list set 416 with the comparison window 414, selects a match window 424 with the highest match score in the candidate list 420, and determines whether the match window 424 references the corrupted data unit 412. If so, the controller performs one or more rounds of the following operations: selecting a comparison window, performing a sliding comparison of the remaining candidate lists, and selecting a match window in the remaining candidate lists. Upon determining that the selected match window (e.g., the third match window 444 in the candidate list 440) does not reference the corrupted data unit 412, the controller replaces the reference to the corrupted data unit 412 in each of the lists 410, 420, 430 (i.e., the lists previously determined to include the corrupted data unit 412) with a reference to the uncorrupted data unit 445 in the candidate subset 444.

[0073] Figure 7 - Example procedure for snapshot repair

[0074] Figure 7 An example process 700 for snapshot repair according to some embodiments is shown. In some examples, a storage controller 110 (eg, Figure 1700 can be performed by a computer program product (such as a processor) or a combination of hardware and programming (e.g., machine-readable instructions that can be executed by a processor). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.

[0075] Block 710 may include identifying, by a storage controller of the deduplication storage system, corrupted data units included in a first snapshot stored in the deduplication storage system. Block 720 may include identifying, by the storage controller, a first manifest referencing the corrupted data units, wherein the first manifest records an ingestion order of the data units of the first snapshot into the deduplication storage system.

[0076] Block 730 may include identifying, by the storage controller, a first container index that indexes the corrupted data unit based on the first manifest. Block 740 may include determining, by the storage controller, a set of candidate manifests based on the first container index. Block 750 may include identifying, by the storage controller, a second manifest having a highest match score with respect to the first portion of the first manifest in the set of candidate manifests.

[0077] Block 760 may include determining, by the storage controller, whether the identified second manifest with the highest match score references a corrupt data unit. Block 770 may include, in response to determining that the identified second manifest with the highest match score does not reference the corrupt data unit, replacing, by the storage controller, a reference to the corrupt data unit in the first manifest with a reference to an uncorrupted data unit referenced in the identified second manifest. Blocks 710-770 may generally correspond to instructions 610-660 (e.g., as referenced above) with reference to FIG. Figure 6 ) described in the example shown.

[0078] Figure 8 - Example machine-readable medium

[0079] Figure 8 A machine-readable medium 800 storing instructions 810-850 is shown according to some embodiments. The instructions 810-860 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 800 may be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium. The instructions 810-860 may generally correspond to the instructions 610-660 (e.g., Figure 6 ) described in the example shown.

[0080] Instructions 810 may be executed to identify corrupted data units included in a first snapshot stored in a deduplication storage system. Instructions 820 may be executed to identify a first manifest referencing the corrupted data units, wherein the first manifest records an ingestion order of data units of the first snapshot into the deduplication storage system.

[0081] Instructions 830 may be executed to identify a first container index that indexes corrupted data units based on the first manifest.Instructions 840 may be executed to determine a set of candidate manifests based on the first container index.

[0082] Instructions 850 may be executed to identify, in the set of candidate manifests, a second manifest having a highest match score with respect to the first portion of the first manifest. Instructions 860 may be executed to, in response to determining that the identified second manifest having the highest match score does not reference the corrupt data unit, replace in the first manifest a reference to the corrupt data unit with a reference to the uncorrupted data unit referenced in the identified second manifest.

[0083] According to some embodiments of the present disclosure, a controller of a deduplication storage system can repair a snapshot including a damaged data unit. The controller can identify the damaged data unit included in the snapshot of the source project. The controller can identify a first list that references the snapshot portion including the damaged data unit. The controller can select a comparison window from the first list, and can compare the comparison window with a set of candidate lists associated with a previous snapshot of the source project. The controller can assign a matching score to each candidate list based on the similarity of each candidate list to the comparison window. If the candidate list with the highest matching score references the damaged data unit, the controller can select a new comparison window from the candidate list, and then the remaining candidate lists can be compared with the new comparison window. When it is determined that the candidate list with the highest score does not reference the damaged data unit, the controller can identify the undamaged data unit from the candidate list, and can determine that the undamaged data unit should be used in the snapshot instead of the damaged data unit. The controller can then replace the reference to the damaged data unit with the reference to the undamaged data unit (in the first list). In this way, the controller can repair the affected list with reduced (or no) data loss, and thus improve the performance of the storage system.

[0084] Note that although Figures 1 to 8 Various examples are shown, but the embodiments are not limited in this regard. Figure 1, it is contemplated that the storage system 100 may include additional devices and / or components, fewer components, different components, different arrangements, etc. In another example, it is contemplated that the functionality of the storage controller 110 described above may be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0085] Data and instructions are stored in corresponding storage devices implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transitory memory, including: semiconductor memory devices, such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks, such as fixed floppy disks and removable disks; other magnetic media, including magnetic tape; optical media, such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices.

[0086] Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly multiple nodes. Such one or more computer-readable or machine-readable storage media are considered to be part of an article (or product). An article or product may refer to a single component or multiple components of any manufacture. One or more storage media may be located in a machine running machine-readable instructions, or at a remote site from which machine-readable instructions may be downloaded over a network for execution.

[0087] In the foregoing description, many details are set forth to enable an understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A computing device, comprising: processor; Memory; as well as a machine-readable storage device storing instructions executable by the processor to: identifying a corrupted data unit included in a first snapshot stored in a deduplication storage system; identifying a first manifest referencing the damaged data unit, wherein the first manifest records an ingestion order of the data units of the first snapshot into the deduplication storage system; a first container index for indexing the damaged data unit based on the first manifest identifier; determining a set of candidate lists based on the first container index; identifying a second inventory in the set of candidate inventory having a highest match score with respect to the first portion of the first inventory; and In response to determining that the identified second manifest does not reference the corrupted data unit, a reference to the corrupted data unit is replaced in the first manifest with a reference to an uncorrupted data unit referenced in the identified second manifest.

2. The computing device of claim 1 , comprising instructions executable by the processor to: A first comparison window is identified in said first list, wherein: The first comparison window is the first portion of the first list; performing a first set of sliding comparisons of the set of candidate lists with the first comparison window; as well as A first set of match scores is calculated for the first comparison window based on the first set of sliding comparisons, wherein each match score in the first set of match scores indicates a similarity between the first comparison window and a matching window in a different candidate list in the set of candidate lists, and wherein the first set of match scores includes the highest match score.

3. The computing device of claim 2, comprising instructions executable by the processor to: In response to determining that the identified second manifest having the highest match score references the corrupted data unit: removing the identified second list from the set of candidate lists to obtain a reduced first set of candidate lists; identifying a second comparison window in the identified second list; performing a sliding comparison of the reduced first candidate list with a second set of the second comparison windows; calculating a second set of match scores for the second comparison window based on the second set of sliding comparisons; identifying a third list in the reduced first set of candidate lists having a highest match score with respect to the second comparison window; as well as In response to determining that the identified third manifest does not reference the corrupted data unit, a reference to the corrupted data unit is replaced in the first manifest with a reference to an uncorrupted data unit referenced in the identified third manifest.

4. The computing device of claim 2, comprising instructions executable by the processor to: For each candidate list in the set of candidate lists, a matching score for each candidate list is calculated based on a total number of non-central data cells referenced in both the each candidate list and the first comparison window.

5. The computing device of claim 2, wherein: The instructions executable to perform the first set of sliding comparisons include instructions executable to: For each sliding comparison in the first set of sliding comparisons: sliding the scanning window across different candidate lists to select different subsets of data units referenced in the different candidate lists; as well as Each different subset of data units is compared to the data units referenced in the first comparison window.

6. The computing device of claim 1, comprising instructions executable by the processor to: After replacing the reference to the corrupted data unit with the reference to the uncorrupted data unit, a first entry in an alias data structure is generated, the first entry indicating that the uncorrupted data unit is a repair replacement for the corrupted data unit.

7. The computing device of claim 6, comprising instructions executable by the processor to: receiving a data unit to be stored in the deduplication storage system; comparing the received data unit to a plurality of entries of the alias data structure; In response to determining that a received data unit matches a first entry of the alias data structure: reading the first entry to identify the uncorrupted data unit; as well as The identified uncorrupted data units are recorded in a manifest that records the order of the received data units.

8. The computing device of claim 1, comprising instructions executable by the processor to: identifying the first manifest based on project metadata representing the first snapshot; and The set of candidate inventories is determined based on a list of inventories stored in the first container index, wherein: The set of candidate lists are arranged in the order of arrival of the list lists.

9. A method comprising: identifying, by a storage controller of the deduplication storage system, a corrupted data unit included in a first snapshot stored in the deduplication storage system; identifying, by the storage controller, a first list referencing the damaged data unit, wherein the first list records an ingestion order of the data units of the first snapshot into the deduplication storage system; a first container index for indexing the damaged data unit based on the first manifest identification by the storage controller; determining, by the storage controller, a set of candidate lists based on the first container index; identifying, by the storage controller, a second inventory in the set of candidate inventories having a highest match score with respect to the first portion of the first inventory; determining, by the storage controller, whether the identified second manifest having the highest match score references the corrupted data unit; and In response to determining that the identified second manifest having the highest match score does not reference the corrupt data unit, replacing, by the storage controller, references to the corrupt data unit in the first manifest with references to uncorrupted data units referenced in the identified second manifest.

10. The method of claim 9, comprising: identifying a first comparison window in the first list, wherein the first comparison window is a first portion of the first list; performing a first set of sliding comparisons of the set of candidate lists with the first comparison window; and Determine a first set of match scores for the first comparison window based on the first set of sliding comparisons, wherein each match score in the first set of match scores indicates a similarity between the first comparison window and a matching window in a different candidate list in the set of candidate lists, and wherein the first set of match scores includes the highest match score.

11. The method of claim 10, comprising: In response to determining that the identified second manifest having the highest match score references the corrupted data unit: removing the identified second list from the set of candidate lists to obtain a reduced first set of candidate lists; identifying a second comparison window in the identified second list; performing a sliding comparison of the reduced first candidate list with a second set of the second comparison windows; determining a second set of match scores for the second comparison window based on the second set of sliding comparisons; identifying a third list in the reduced first set of candidate lists having a highest match score with respect to the second comparison window; as well as In response to determining that the identified third manifest does not reference the corrupted data unit, a reference to the corrupted data unit is replaced in the first manifest with a reference to an uncorrupted data unit referenced in the identified third manifest.

12. The method of claim 10, comprising: For each candidate list in the set of candidate lists, a match score for each candidate list is determined based on a total number of non-central data cells referenced in both the each candidate list and the first comparison window.

13. The method of claim 9, comprising: After replacing the reference to the corrupted data unit with the reference to the uncorrupted data unit, a first entry in an alias data structure is generated, the first entry indicating that the uncorrupted data unit is a repair replacement for the corrupted data unit.

14. The method of claim 13, comprising: receiving a data unit to be stored in the deduplication storage system; comparing the received data unit to a plurality of entries of the alias data structure; In response to determining that a received data unit matches a first entry of the alias data structure: reading the first entry to identify the uncorrupted data unit; as well as The identified uncorrupted data units are recorded in a manifest that records the order of the received data units.

15. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to: identifying a corrupted data unit included in a first snapshot stored in a deduplication storage system; identifying a first manifest referencing the corrupted data unit, wherein: The first list records the ingestion order of data units of the first snapshot into the deduplication storage system; a first container index for indexing the damaged data unit based on the first manifest identifier; determining a set of candidate lists based on the first container index; identifying, from the set of candidate lists, a second list having a highest match score with respect to the first portion of the first list; as well as In response to determining that the identified second manifest having the highest match score does not reference the corrupted data unit, replacing the reference to the corrupted data unit in the first manifest with a reference to an uncorrupted data unit referenced in the identified second manifest.

16. The non-transitory machine-readable medium of claim 15, comprising instructions that when executed cause the processor to: A first comparison window is identified in said first list, wherein: The first comparison window is the first portion of the first list; performing a first set of sliding comparisons of the set of candidate lists with the first comparison window; as well as A first set of match scores is calculated for the first comparison window based on the first set of sliding comparisons, wherein each match score in the first set of match scores indicates a similarity between the first comparison window and a matching window in a different candidate list in the set of candidate lists, and wherein the first set of match scores includes the highest match score.

17. The non-transitory machine-readable medium of claim 16, comprising instructions that when executed cause the processor to: In response to determining that the identified second manifest having the highest match score references the corrupted data unit: removing the identified second list from the set of candidate lists to obtain a reduced first set of candidate lists; identifying a second comparison window in the identified second list; performing a sliding comparison of the reduced first candidate list with a second set of the second comparison windows; calculating a second set of match scores for the second comparison window based on the second set of sliding comparisons; identifying a third list in the reduced first set of candidate lists having a highest match score with respect to the second comparison window; as well as In response to determining that the identified third manifest does not reference the corrupted data unit, a reference to the corrupted data unit is replaced in the first manifest with a reference to an uncorrupted data unit referenced in the identified third manifest.

18. The non-transitory machine-readable medium of claim 16, comprising instructions that when executed cause the processor to: For each candidate list in the set of candidate lists, a matching score for each candidate list is calculated based on a total number of non-central data cells referenced in both the each candidate list and the first comparison window.

19. The non-transitory machine-readable medium of claim 16, comprising instructions that when executed cause the processor to: For each sliding comparison in the first set of sliding comparisons: Sliding the scanning window across different candidate lists to select different subsets of data units referenced in the different candidate lists; and Each different subset of data units is compared to the data units referenced in the first comparison window.

20. The non-transitory machine-readable medium of claim 15, comprising instructions that when executed cause the processor to: identifying the first manifest based on project metadata representing the first snapshot; and The set of candidate inventories is determined based on a list of inventories stored in the first container index, wherein: The set of candidate lists are arranged in the order of arrival of the list lists.

Images