Storage location metadata for deduplication storage systems

Abstract

The invention discloses storage location metadata for a deduplication storage system. Example implementations relate to data deduplication operations in a storage system. Examples include receiving a stream of data units to be stored in persistent storage of a deduplication storage system; generating a list to record a receiving sequence of the data units; the data units are indexed into container indexes, and each container index records fingerprints and immutable storage positions of one or more data units; in response to determining that the manifest has reached the threshold size, determining an immutable storage location recorded in the container index; and updating the manifest to record the determined immutable storage location.

Description

Background Art

[0001] Data reduction techniques may be applied to reduce the amount of data stored in a storage system. Example data reduction techniques include data deduplication. Data deduplication identifies duplicate data units and seeks to reduce or eliminate the number of instances of the duplicate data units stored in a storage system. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Some embodiments are described with respect to the following figures.

[0003] Figure 1 is a schematic diagram of an example storage system according to some implementations.

[0004] FIG. 2A to FIG. 2B is an illustration of an example data structure according to some implementations.

[0005] Figure 3 is an illustration of an example process according to some implementations.

[0006] Figure 4 is an illustration of an example process according to some implementations.

[0007] Figure 5 is a schematic diagram of an example computing device in accordance with some implementations.

[0008] Figure 6 is an illustration of an example process according to some implementations.

[0009] Figure 7 is a diagram of an example machine-readable medium having instructions stored thereon, according to some implementations.

[0010] Throughout the drawings, the same reference numerals refer to similar, but not necessarily identical, elements. The drawings are not necessarily drawn to scale, and the size of some parts may be exaggerated to more clearly illustrate the examples shown. In addition, the drawings provide examples and / or implementations consistent with the description; however, the description is not limited to the examples and / or implementations provided in the drawings. DETAILED DESCRIPTION

[0011] In the present disclosure, unless the context clearly indicates otherwise, the use of the terms "a", "an" or "the" is intended to include the plural forms as well. Similarly, when used in the present disclosure, the terms "includes / including / comprises / comprising" or "have / having" specify the presence of the elements, but do not exclude the presence or addition of other elements.

[0012] In some examples, a storage system may back up a collection of data (referred to herein as a "stream" or "data stream" of data) in a deduplicated form, thereby reducing the amount of storage space required to store the data stream. The storage system may create a "backup item" to represent the data stream in a deduplicated form. The storage system may perform a deduplication process including breaking the data stream into discrete data units (or "chunks") and determining "fingerprints" of these incoming data units (as described below). Further, the storage system may compare the fingerprints of the incoming data units with the fingerprints of the stored data units, and may thereby determine which incoming data units are duplicates of previously stored data units (e.g., when the comparison indicates matching fingerprints). In the event that the data units are duplicates, the storage system may store a reference to the previously stored data unit instead of storing the duplicate incoming data unit. The process for receiving and deduplicating an incoming data stream may be referred to herein as a "data ingestion" process of the storage system.

[0013] As used herein, "fingerprint" refers to a value obtained by applying a function to the contents of a data unit (where "content" may include all or a subset of the contents of a data unit). Examples of functions that may be applied include hash functions that produce a hash value based on the contents of an incoming data unit. Examples of hash functions include cryptographic hash functions, such as a Secure Hash Algorithm 2 (SHA-2) hash function (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions may be employed.

[0014] A "storage system" may include a storage device or an array of storage devices. A storage system may also include (multiple) storage controllers that manage access to (multiple) storage devices. A "data unit" may refer to any data portion that can be individually identified in a storage system. In some cases, a data unit may refer to a block, a collection of blocks, or any other data portion. In some examples, a storage system may store data units in a persistent storage device. A persistent storage device may be implemented using one or more (multiple) persistent (e.g., non-volatile) storage devices (such as (multiple) disk-based storage devices (e.g., (multiple) hard disk drives (HDDs)), (multiple) solid-state devices (SSDs) (such as (multiple) flash storage devices), etc. or a combination thereof. A "controller" may refer to a hardware processing circuit, which may include any one or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuits. Alternatively, a "controller" may refer to a combination of a hardware processing circuit and machine-readable instructions (software and / or firmware) that can be executed on the hardware processing circuit.

[0015] In some examples, a deduplication storage system may use metadata to process an inbound data stream (e.g., a backup project). For example, such metadata may include a data recipe (also referred to herein as a "manifest") that specifies the order in which specific data units of each backup project are received. Further, such metadata may include project metadata to represent each received backup project (e.g., a data stream) in a deduplicated form. The project metadata may include an identifier for a set of manifests and may indicate the order of the set of manifests. Processing each backup project may be referred to herein as a "backup process." Subsequently, in response to a read request, the deduplication system may use the project metadata and the set of manifests to determine the order in which the data units are received, so that the original data stream of the backup project can be recreated. Accordingly, the set of manifests may be a representation of the original backup project. The manifest may include a series of records, each record representing a specific set (multiple) of data units. The records in the manifest may include one or more fields identifying container indexes that index the data units (e.g., include storage information of the data units). For example, the container index may include one or more fields that specify location information (e.g., container, offset, etc.) of the stored data units, compression and / or encryption characteristics of the stored data units, and the like. Further, the container index may include reference counts indicating the number of manifests that reference each data unit.

[0016] In some examples, when a data unit (e.g., in a data stream) is received, it can be matched with one or more container indexes to determine whether the same data unit is already stored in a container of a deduplication storage system. For example, the deduplication storage system can compare the fingerprint of the received data unit with the fingerprint in one or more container indexes. If a matching fingerprint is not found in the searched container index (multiple), the received data unit can be added to the container, and the entry of the received data unit can be added to the container index corresponding to the container. However, if a matching fingerprint is found in the searched container index, it can be determined that the same data unit as the received data unit is already stored in the container. In response to this determination, the reference count of the corresponding entry is incremented, and the received data unit is not stored in the container (because it already exists in one of the containers), thereby avoiding the storage of duplicate data units in the deduplication storage system. As used herein, the term "matching operation" can refer to an operation for comparing the fingerprint of a set of multiple data units (e.g., from a specific backup data stream) with the fingerprint stored in the container index.

[0017] In some examples, a deduplication storage system may store data units and associated metadata objects in a remote storage device (e.g., a "cloud" or network storage service) rather than in a local file system. In such an example, processing a read request for a stored data unit may include: loading an appropriate manifest from the remote storage device into memory, reading the manifest to identify an appropriate container index, loading the container index from the remote storage device into memory, reading the container index to determine a location in the remote storage device where the data unit is stored, and then loading the data unit from the location in the remote storage device into memory. Thus, executing the read request may include input / output (I / O) operations for loading the manifest, container index, and data unit. Therefore, completing the read request may consume network bandwidth and affect the processing performance of the deduplication storage system.

[0018] According to some embodiments of the present disclosure, a controller of a deduplication storage system may assign an immutable storage location to a data unit at the time of initial ingestion (e.g., when first received and stored), and may record the immutable storage location in one or more container indexes loaded in a memory. As used herein, an "immutable storage location" of a data unit may refer to a storage address that does not change after being assigned to a data unit. The controller may record the order in which the data unit is received in a list loaded in the memory. In some embodiments, when the information recorded in the list reaches a threshold level (e.g., a maximum level), the controller may iteratively traverse (multiple) container indexes to extract the immutable storage location of the data unit. When the data ingestion process is completed (e.g., the backup process is completed), the controller may load the container index and the list from the memory into a persistent storage device, and may delete these items from the memory. Subsequently, in order to access the stored data unit (e.g., in response to a read request), the controller may load the appropriate list from the persistent storage device into the memory. The controller may read the list to determine the immutable storage location of the data unit, and then may access the data unit using the immutable storage location. The controller can perform the read operation without having to load the container index from the storage device into the memory (i.e., determine the storage address of the data unit being accessed). In this way, the number of data transfers from the storage device to the memory can be reduced. Therefore, some embodiments can improve the performance of the deduplication storage system. Figures 1 to 7 Various aspects of the disclosed technology are further discussed.

[0019] Figure 1 - Sample storage system

[0020] Figure 1An example system 105 is shown that includes a storage system 100 and a remote storage device 190. According to some embodiments, the storage system 100 may include a storage controller 110, a memory 115, and a persistent storage device 140. The storage system 100 may be coupled to the remote storage device 190 via a network connection. The remote storage device 190 may be a network-based persistent storage facility or service (also referred to herein as a "cloud-based storage device"). In some examples, the use of the remote storage device 190 may incur financial charges based on the number of separate transfers.

[0021] The persistent storage device 140 may include one or more non-transitory storage media, such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc., or a combination thereof. The memory 115 may be implemented with a semiconductor memory such as a random access memory (RAM). In some examples, the storage controller 110 may be implemented via hardware (e.g., an electronic circuit) or a combination of hardware and programming (e.g., including at least one processor and instructions executable by the at least one processor and stored on at least one machine-readable storage medium). In some embodiments, the memory 115 may include a list 150 and a container index 160. Further, the persistent storage device 140 may store the list 150 and the container index 160. The remote storage device 190 may persistently store a container entity group (CEG) object 170. Each CEG object 170 may be a container data structure configured to store multiple data units.

[0022] In some embodiments, the storage system 100 can perform deduplication on stored data. For example, the storage controller 110 can divide the input data stream into data units, and can include at least one copy of each data unit in at least one CEG object 170. The storage controller 110 can generate a manifest 150 to record the order in which the data units in the data stream are received. The manifest 150 may include a pointer or other information indicating a container index 160 associated with each data unit. In some embodiments, the container index 160 may include a fingerprint (e.g., a hash) of the stored data unit for use in a matching process of the deduplication process. The container index 160 may also include a reference count (e.g., indicating the number of manifests 150 that reference each data unit) for the data unit used in housekeeping (e.g., for determining whether to delete the stored data unit). Reference is made below to FIG. 2A to FIG. 2B Example implementations of manifest 150, container index 160, and CEG object 170 are discussed further.

[0023] In some embodiments, the container index 160 may include address information 155 for use in the reconstruction of deduplicated data, the address information being used to record or indicate the storage location of the data unit. Further, in some embodiments, the manifest 150 may include address information 155 for each data unit referenced by the manifest 150. For example, the address information 155 (e.g., stored in the container index 160 and the manifest 150) may include an immutable address identifying the CEG object 170 storing the data unit and the location (within the CEG object 170) where the data unit is stored. Figure 3 An example process for storing address information 155 in manifest 150 is discussed.

[0024] In some embodiments, the storage controller 110 may receive a read request to access the stored data, and in response may access the manifest 150 to determine the sequence of data units that make up the original data. Further, the storage controller 110 may use the address information 155 included in the manifest 150 to determine the location where the data units are stored (e.g., for each data unit, the corresponding CEG object 170, the offset, etc.). In this way, the storage controller 110 may determine the storage location of the data unit without loading the container index 160 into the memory 115. The storage controller 110 may then read the data unit from the determined location (e.g., by transferring the data unit from the remote storage device 190 to the memory 115. Reference is made to Figure 4 An example process for performing a read request using address information 155 in manifest 150 is discussed.

[0025] Note that although Figure 1 An example is shown, but embodiments are not limited in this regard. For example, it is contemplated that some or all of manifest 150 and container index 160 may be stored in remote storage 190. In another example, it is contemplated that some or all of CEG objects 170 may be stored in persistent storage 140. In yet another example, it is contemplated that memory 115, persistent storage 140, and / or remote storage 190 may include other data objects or metadata. Further, it is contemplated that storage system 100 may include additional devices and / or components, fewer components, different components, different arrangements, etc.

[0026] FIG. 2A to FIG. 2B - Example data structure

[0027] Figure 2A2 shows an example data structure 200 for use in deduplication according to some embodiments. As shown, data structure 200 may include item metadata 202, manifest 203, container index 220, and container entity group ("CEG") object 250. In some examples, manifest 203, container index 220, and CEG object 250 may generally correspond to manifest 150, container index 160, and CEG object 170, respectively (e.g., Figure 1 Further, in some examples, the data structure 200 may be stored by the storage controller 110 (eg, Figure 1 as shown) to generate and / or manage.

[0028] In some implementations, the item metadata 202 may include multiple manifest identifiers 205. Each manifest identifier 205 may identify a different manifest 203. In some implementations, the manifest identifiers 205 may be arranged in a stream order (i.e., based on the order in which the data units represented by the identified manifest 203 are received). Further, although for simplicity Figure 2A , only one of each data structure is shown, but the data structure 200 may include multiple instances of project metadata 202, each instance including or pointing to one or more manifests 203. In such an example, the data structure 200 may include multiple manifests 203. The manifests 203 may include multiple manifest records 210 that reference multiple container indexes 220. Each container index 220 may include multiple unit metadata 225. Each instance of the unit metadata 225 may index one or more data units 260. Each CEG object 250 may include multiple data units 260. Further, in some examples, the CEG object 250 may include one or more groups or "entities" 255, wherein each entity 255 includes multiple data units 260.

[0029] Reference now Figure 2B, showing container index metadata 270 and manifest metadata 280. In some embodiments, manifest metadata 280 may be included in manifest 203 (e.g., in manifest record 210). Further, container index metadata 270 may be included in container index 220 (e.g., in cell metadata 225). In some embodiments, container index metadata 270 and manifest metadata 280 may each include a cell address, a cell length, and compression information. A cell address may be information stored in a field (or a combination of multiple fields) that deterministically identifies a storage location of one or more data units. Further, a cell length may specify a data length of (multiple) data units stored at a cell address. In some embodiments, a cell address of a data unit may be recorded as three values ​​(e.g., stored in three fields), which respectively identify a specific CEG object 250, a specific entity 255, and an offset within the specific entity 255. In other embodiments, a cell address may be an indication of a cell added to a deduplication storage system (e.g., Figure 1 A numerical value (referred to as an "arrival number") is a numerical value (referred to as an "arrival number") that represents the order of arrival (also referred to as an "ingestion order") of a data unit in the system 105 shown in FIG. 10A . In some implementations, the unit address in manifest metadata 280 does not change after being populated in manifest metadata 280, and thus may represent an immutable address of a data unit.

[0030] In some embodiments, the container index metadata 270 and / or the manifest metadata 280 may use a run-length reference format to represent a continuous range of data units (e.g., a portion of a data stream) stored within a single CEG object 250 (or within a single entity 255). For example, the unit address field may record the offset (in the CEG object 250) of the start of the first data unit in the represented data range, and the unit length field may indicate the length of the represented data range. In another example, the unit address field may record the arrival number of the first data unit in the represented data unit range, and the unit length field may indicate the number N (where "N" is an integer) of data units in the data unit range that are after the first data unit specified by the arrival number in the unit address field. The data units in the data unit range may have consecutive arrival numbers (e.g., because they are continuous in the ingested data stream). In this way, the data unit range may be represented by the arrival number of the first data unit in the data unit range (e.g., recorded in the unit address field) and the number N of additional data units in the data unit range (e.g., recorded in the unit length field). Additional data units in the data unit range that follow the first data unit may be deterministically obtained by calculating the N arrival numbers that are sequentially subsequent to the designated arrival number of the first data unit, where the N arrival numbers identify additional data units in the data unit range. For example, the manifest metadata 280 may include an arrival number "X" in the unit address field and a number N in the unit length field to indicate a data unit range that includes a first data unit specified by the arrival number X and subsequent data units specified by arrival numbers X+i, where i=1 to i=N (inclusive) (where "i" is an integer). In this manner, the run length reference format may be used to identify all data units in the data unit range.

[0031] In some implementations, the compression information may indicate how the stored data unit was compressed or decompressed (whether compression was used, the type of compression code, the type of decompression code, the decompressed size, a checksum value, etc.) In some examples, during a read operation, the compression information may be used to decompress the requested data unit (or a particular entity 255 that includes the requested data unit).

[0032] In some embodiments, the container index metadata 270 may include a fingerprint and a reference count. The fingerprint may be a value obtained by applying a function (e.g., a hash function) to all or part of the contents of a data unit. The reference count may indicate the total number of manifest records 210 that reference the data unit. Further, in some embodiments, the fingerprint and the reference count may not be included in the manifest metadata 280.

[0033] Note that although FIG. 2A to FIG. 2B An example of data structure 200 is shown, but embodiments are not limited in this regard. For example, it is contemplated that project metadata 202, manifest 203, container index 220, and CEG object 250 may include additional fields or elements, additional data structures, etc. In another example, it is contemplated that container index metadata 270 and / or manifest metadata 280 may include additional fields or elements.

[0034] Figure 3 - Example process for generating metadata

[0035] Figure 3 An example process 300 for generating metadata for a deduplication storage system according to some embodiments is shown. In some examples, a storage controller 110 (eg, Figure 1 300). Process 300 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 300 may be executed by a single processing thread. In other embodiments, process 300 may be executed in parallel by multiple processing threads (e.g., using a work map and executing multiple housekeeping jobs simultaneously).

[0036] Block 310 may include receiving a data unit to be stored in a persistent storage device of a deduplication storage system. Block 315 may include generating a fingerprint for the received data unit. Block 320 may include comparing the generated fingerprint to a fingerprint stored in a container index of the deduplication storage system. For example, referring to Figure 1 , storage controller 110 receives a stream of data units (eg, backup items) to be stored in system 105 . Storage controller 110 generates fingerprints for the received data units and compares the generated fingerprints with fingerprints included in container index 160 .

[0037] Decision block 325 may include determining whether a match is found between the generated fingerprint and a fingerprint included in the container index (CI). If it is determined at decision block 325 that a match is not found ("No"), process 300 may continue at block 330, including assigning an immutable storage location to the data unit. Block 335 may include storing the data unit in the assigned immutable storage location. Block 340 may include recording the fingerprint and immutable storage location of the data unit in the container index. Block 345 may include recording the identifier of the data unit and the container index in the manifest. For example, reference Figure 1If no match is identified between the generated fingerprint of the received data unit and the fingerprint included in the container index 160, the storage controller 110 stores the data unit in the immutable storage location in the new CEG object 170. The storage controller 110 adds a new entry for the data unit to the container index 160. The new entry of the container index 160 may store the container index metadata 270 (e.g., Figure 2B As shown), the container index metadata includes the fingerprint, reference count, unit address, unit length and compression information of the data unit.

[0038] However, if it is determined at decision block 325 that a match is found ("yes"), process 300 may continue at block 350, including updating a reference count of the data unit in the container index that indexes the data unit. After block 350, process 300 may continue at block 345, including recording the identifiers of the data unit and the container index in the manifest. For example, reference Figure 1 If a match is identified between the generated fingerprint of the received data unit and the fingerprint stored in the specific container index 160, the storage controller 110 determines that a duplicate of the data unit is already stored in the existing CEG object 170. Accordingly, the storage controller 110 increments the reference count (corresponding to the data unit) in the specific container index 160. Further, the storage controller 110 records the information identifying the data unit and the specific container index 160 in the current manifest 150.

[0039] Decision block 360 may include determining whether the manifest is full (e.g., whether a maximum number of entries representing a data unit has been reached). If not ("No"), process 300 may return to block 310 (i.e., receive another data unit to be stored in persistent storage). In other cases, if it is determined at decision block 360 that the manifest is full ("Yes"), process 300 may continue at block 370, including reading each container index recorded in the manifest to determine an immutable storage location of a data unit recorded in the manifest. Block 380 may include recording the determined immutable storage location of the data unit in the manifest. For example, referring to Figure 1 Upon determining that the current manifest 150 is full, the storage controller 110 accesses one or more container indexes 160 (e.g., loaded in the memory 115) that index a group of data units recorded in the current manifest 150, and reads the container indexes 160 to obtain the address information 115 indicating the immutable storage location of the group of data units. The storage controller 115 then stores the address information 115 of the group of data units in the manifest 150 (e.g., in the manifest metadata 280 including the unit address, unit length, and compression information).

[0040] In some embodiments, the address information 155 of the range of consecutive data units can be represented in a run-length reference format. For example, if the range of consecutive data units is stored unchanged within a single CEG object 250 (or within a single entity 255), the storage controller 110 can use a unit address field (storing the offset or arrival number of the first data unit in the range) and a unit length field (storing the number of data units after the first data unit in the range) to record the address information 155 of the range. Subsequently, the address information 115 stored in the manifest 150 can be used to perform a read request without having to load the container index 160 into the memory 115. hereinafter, reference is made to Figure 4 An example process for performing a read request using address information 155 stored in manifest 150 is discussed.

[0041] Block 390 may include initializing a new manifest. After block 390, process 300 may return to block 310 (i.e., receiving another data unit to be stored in persistent storage). Alternatively, process 300 may be completed. For example, referring to Figure 1 After storing the immutable storage locations of the identified set of data units in the manifest 150, the storage controller 110 initiates a new manifest 150 to record information about subsequently received data units.

[0042] Figure 4 - Example procedure for performing a read request

[0043] Figure 4 An example process 400 for executing a read request in a deduplication storage system according to some embodiments is shown. In some examples, a storage controller 110 (eg, Figure 1 400). Process 400 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 400 may be executed by a single processing thread. In other embodiments, process 400 may be executed in parallel by multiple processing threads (e.g., using a work map and executing multiple housekeeping jobs simultaneously).

[0044] Block 410 may include receiving a read request for a data unit stored in a deduplication storage system. Block 420 may include identifying a list of recorded data units. Block 430 may include loading the identified list into a memory. For example, referring to Figure 1, storage controller 110 may receive a read request to access a data unit stored in remote storage 190 (e.g., as part of a stored backup project). Storage controller 110 identifies a manifest 150 (in persistent storage 140) that references the requested data unit and loads manifest 150 into memory 115. Further, in some examples, a container index 160 that indexes the requested data unit is not loaded into memory 115 in response to the read request.

[0045] Block 440 may include obtaining an immutable storage location of a data unit from a manifest loaded in memory. Block 450 may include reading a data unit according to the immutable storage location obtained from the manifest loaded in memory. After block 450, process 400 may be complete. For example, referring to Figure 1 , the storage controller 110 reads the address information 155 stored in the manifest 150 (which is loaded in the memory 115) to determine the immutable storage location of the requested data unit. The storage controller 110 then reads the requested data unit from the immutable storage location (e.g., in the CEG object 170 stored in the remote storage device 190). In this way, the storage controller 110 performs the read request without having to load the container index 160 from the persistent storage device 140 (or the remote storage device 190) into the memory 115. Therefore, the process 400 can reduce the number of input / output (I / O) transfers required to perform the read request, and thus can improve the performance of the storage system 100.

[0046] Figure 5 - Sample computing device

[0047] Figure 5 1 shows a schematic diagram of an example computing device 500. In some examples, computing device 500 may generally correspond to storage system 100 (eg, Figure 1 As shown, computing device 500 may include hardware processor 502, memory 504, and machine-readable storage device 505 including instructions 510-560. Machine-readable storage device 505 may be a non-transitory medium. Instructions 510-560 may be executed by hardware processor 502 or by a processing engine included in hardware processor 502.

[0048] Instructions 510 may be executed to receive a plurality of data units to be stored in a persistent storage device of a deduplication storage system. Instructions 520 may be executed to generate a manifest to record the order in which the plurality of data units are received. For example, referring to Figure 1, the storage controller 110 receives a stream of data units (eg, backup items) to be stored in the deduplication storage system 100 and records the ingestion order of the data units in the current manifest 150 .

[0049] Instructions 530 may be executed to index the plurality of data units into a plurality of container indexes, wherein each container index records the fingerprint and immutable storage location of one or more data units. Figure 1 , the storage controller 110 generates a fingerprint for the received data unit and compares the generated fingerprint with the fingerprint included in the container index 160. If no match is identified between the generated fingerprint of the received data unit and the fingerprint included in the container index 160, the storage controller 110 stores the data unit in the immutable storage location in the new CEG object 170. Further, the storage controller 110 records the information identifying the fingerprint of the data unit and the immutable storage location in the container index 160. In other cases, if a match is identified between the generated fingerprint of the data unit and the fingerprint included in the container index 160, the storage controller 110 determines that a duplicate of the data unit is already stored in the existing CEG object 170. Accordingly, the storage controller 110 increments a reference count (corresponding to the data unit) in the specific container index 160. Further, the storage controller 110 records the information identifying the data unit and the specific container index 160 in the current manifest 150.

[0050] Instructions 540 may be executed to determine a plurality of immutable storage locations recorded in a plurality of container indexes in response to determining that the manifest has reached a threshold size. Instructions 550 may be executed to update the manifest to record the determined plurality of immutable storage locations. For example, referring to Figure 1 Upon determining that the current manifest 150 is full, the storage controller 110 accesses one or more container indexes 160 (e.g., loaded in the memory 115) that index a group of data units recorded in the current manifest 150, and reads the container indexes 160 to obtain the address information 115 indicating the immutable storage location of the group of data units. The storage controller 115 then stores the address information 115 of the group of data units in the manifest 150 (e.g., in the manifest metadata 280 including the unit address, unit length, and compression information).

[0051] Figure 6 - Example process for generating metadata

[0052] Figure 6 An example process 600 for generating metadata for a deduplication storage system according to some embodiments is shown. In some examples, a storage controller 110 (eg, Figure 1600 is performed by a computer program product (shown in FIG. 600 ). The process 600 may be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions may be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. The machine-readable instructions may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.

[0053] Block 610 may include receiving, by a storage controller of a deduplication storage system, a plurality of data units to be stored in a persistent storage device of the deduplication storage system. Block 620 may include generating, by the storage controller, a manifest to record an order in which the plurality of data units were received. Block 630 may include indexing, by the storage controller, the plurality of data units into a plurality of container indexes, wherein each container index records a fingerprint and an immutable storage location of one or more data units.

[0054] Block 640 may include determining, by the storage controller, whether the manifest has reached a threshold size. Block 650 may include, in response to determining that the manifest has reached the threshold size, determining, by the storage controller, a plurality of immutable storage locations recorded in the plurality of container indexes. Block 660 may include updating, by the storage controller, the manifest to record the determined plurality of immutable storage locations. Blocks 610-660 may generally correspond to instructions 510-550 (e.g., as referenced above) with reference to FIG. Figure 5 ) described in the example shown.

[0055] Figure 7 - Example machine-readable medium

[0056] Figure 7 A machine-readable medium 700 storing instructions 710-750 according to some embodiments is shown. The instructions 710-750 may be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 700 may be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium. The instructions 710-750 may generally correspond to the instructions 510-550 (e.g., the instructions 510-550 described above). Figure 5 ) described in the example shown.

[0057] Instructions 710 may be executed to receive a plurality of data units to be stored in a persistent storage device of a deduplication storage system. Instructions 720 may be executed to generate a manifest to record the order in which the plurality of data units were received. Instructions 730 may be executed to index the plurality of data units into a plurality of container indexes, wherein each container index records a fingerprint and an immutable storage location of one or more data units.

[0058] Instructions 740 may be executed to determine a plurality of immutable storage locations recorded in a plurality of container indexes in response to determining that the manifest has reached a threshold size. Instructions 750 may be executed to update the manifest to record the determined plurality of immutable storage locations.

[0059] According to some embodiments of the present disclosure, a controller of a deduplication storage system may assign an immutable storage location to a data unit upon initial ingestion, and may record the immutable storage location in one or more container indexes loaded into a memory. In some embodiments, when the information recorded in the manifest reaches a threshold level, the controller may iteratively traverse (multiple) container indexes to extract the immutable storage location of the data unit. Subsequently, in response to a read request, the controller may load the appropriate manifest from a persistent storage device into the memory. The controller may read the manifest to determine the immutable storage location of the data unit, and may then access the data unit using the immutable storage location. The controller may perform a read operation without having to load the container index into the memory. In this way, the number of data transfers from the storage device to the memory may be reduced. Therefore, some embodiments may improve the performance of a deduplication storage system.

[0060] Note that although Figures 1 to 7 Various examples are shown, but the embodiments are not limited in this regard. Figure 1 It is contemplated that the functionality of the above-described storage controller 110 may be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0061] Data and instructions are stored in corresponding storage devices implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transitory memory, including: semiconductor memory devices, such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks, such as fixed floppy disks and removable disks; other magnetic media, including magnetic tape; optical media, such as compact disks (CDs) or digital video disks (DVDs); or other types of storage devices.

[0062] Note that the instructions discussed above may be provided on one computer-readable or machine-readable storage medium, or alternatively, may be provided on multiple computer-readable or machine-readable storage media distributed in a large system having possibly multiple nodes. Such one or more computer-readable or machine-readable storage media are considered to be part of an article (or product). An article or product may refer to a single component or multiple components of any manufacture. One or more storage media may be located in a machine running machine-readable instructions, or at a remote site from which machine-readable instructions may be downloaded over a network for execution.

[0063] In the foregoing description, many details are set forth to enable an understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A computing device, comprising: processor; Memory; as well as A machine-readable storage device storing instructions executable by the processor to: receiving a plurality of data units to be stored in a persistent storage device of a deduplication storage system; generating a list to record the order in which the plurality of data units are received; Indexing the plurality of data units into a plurality of container indexes, wherein each container index records fingerprints and immutable storage locations of one or more data units; In response to determining that the manifest has reached a threshold size, determining a plurality of immutable storage locations recorded in the plurality of container indexes; and The manifest is updated to record the determined plurality of immutable storage locations.

2. The computing device of claim 1 , comprising instructions executable by the processor to: receiving a read request for a particular data unit stored in the persistent storage device; identifying a particular manifest based on the read request for the particular data unit; loading the specific manifest into the memory; obtaining an immutable storage location of said particular data unit from said particular manifest loaded in said memory; as well as The specific data unit is read according to the immutable storage location obtained from the specific manifest loaded into the memory.

3. The computing device of claim 2, wherein: The particular data unit is indexed by a particular container index, and wherein the particular container index is not loaded into the memory to perform a read request for the particular data unit.

4. The computing device of claim 1 , comprising instructions executable by the processor to: generating a first fingerprint for a first data unit included in the received plurality of data units; comparing the first fingerprint to fingerprints stored in the plurality of container indexes; In response to determining that no match is found between the first fingerprint and the fingerprints stored in the plurality of container indexes: assigning a first immutable storage location to the first data unit; storing the first data unit in the first immutable storage location; recording the first fingerprint and the first immutable storage location in a first container index; as well as Identifiers of the first data unit and the first container index are recorded in the manifest.

5. The computing device of claim 4, comprising instructions executable by the processor to: In response to determining that a match is found between the first fingerprint and a fingerprint stored in a second container index: updating a reference count of the first data unit in the second container index; and Identifiers of the first data unit and the second container index are recorded in the manifest.

6. The computing device of claim 1 , comprising instructions executable by the processor to: After updating the manifest to record the determined plurality of immutable storage locations, a new manifest is initialized.

7. The computing device of claim 1, comprising instructions executable by the processor to: In response to determining that a range of contiguous data units is stored within a single data object, a run length reference value is recorded in the manifest to indicate an immutable storage location of the range of contiguous data units within the single data object.

8. The computing device of claim 7, wherein: The single data object is a container entity group ("CEG") object stored on a cloud-based remote storage device.

9. A method comprising: Receiving, by a storage controller of the deduplication storage system, a plurality of data units to be stored in a persistent storage device of the deduplication storage system; generating, by the storage controller, a list to record the order in which the plurality of data units are received; The storage controller indexes the plurality of data units into a plurality of container indexes, wherein each container index records fingerprints and immutable storage locations of one or more data units; determining, by the storage controller, whether the list has reached a threshold size; In response to determining that the manifest has reached a threshold size, determining, by the storage controller, a plurality of immutable storage locations recorded in the plurality of container indexes; and The manifest is updated, by the storage controller, to record the determined plurality of immutable storage locations.

10. The method of claim 9, comprising: receiving a read request for a particular data unit stored in the persistent storage device; identifying a specific manifest based on a read request for the specific data unit; loading the specific manifest into the memory; obtaining an immutable storage location of said particular data unit from said particular manifest loaded in said memory; as well as The specific data unit is read according to the immutable storage location obtained from the specific manifest loaded into the memory.

11. The method of claim 10, wherein: The particular data unit is indexed by a particular container index, and wherein the particular container index is not loaded into the memory to perform a read request for the particular data unit.

12. The method of claim 9, comprising: generating a first fingerprint for a first data unit included in the received plurality of data units; matching the first fingerprint with fingerprints stored in the plurality of container indexes; In response to determining that no match is found between the first fingerprint and the fingerprints stored in the plurality of container indexes: assigning a first immutable storage location to the first data unit; storing the first data unit in the first immutable storage location; recording the first fingerprint and the first immutable storage location in a first container index; as well as Identifiers of the first data unit and the first container index are recorded in the manifest.

13. The method of claim 12, comprising: In response to determining that a match is found between the first fingerprint and a fingerprint stored in a second container index: updating a reference count of the first data unit in the second container index; as well as Identifiers of the first data unit and the second container index are recorded in the manifest.

14. The method of claim 9, comprising: After updating the manifest to record the determined plurality of immutable storage locations, a new manifest is initialized.

15. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to: receiving a plurality of data units to be stored in a persistent storage device of a deduplication storage system; generating a list to record the order in which the plurality of data units are received; Indexing the plurality of data units into a plurality of container indexes, wherein each container index records fingerprints and immutable storage locations of one or more data units; In response to determining that the manifest has reached a threshold size, determining a plurality of immutable storage locations recorded in the plurality of container indexes; and The manifest is updated to record the determined plurality of immutable storage locations.

16. The non-transitory machine-readable medium of claim 15, comprising instructions that when executed cause the processor to: receiving a read request for a particular data unit stored in the persistent storage device; identifying a particular manifest based on the read request for the particular data unit; loading the particular manifest into a memory; obtaining an immutable storage location of said particular data unit from said particular manifest loaded in said memory; as well as The specific data unit is read according to the immutable storage location obtained from the specific manifest loaded into the memory.

17. The non-transitory machine-readable medium of claim 16, wherein: The particular data unit is indexed by a particular container index, and wherein the particular container index is not loaded into the memory to perform a read request for the particular data unit.

18. The non-transitory machine-readable medium of claim 15, comprising instructions that when executed cause the processor to: generating a first fingerprint for a first data unit included in the received plurality of data units; comparing the first fingerprint to fingerprints stored in the plurality of container indexes; In response to determining that no match is found between the first fingerprint and the fingerprints stored in the plurality of container indexes: assigning a first immutable storage location to the first data unit; storing the first data unit in the first immutable storage location; recording the first fingerprint and the first immutable storage location in a first container index; as well as Identifiers of the first data unit and the first container index are recorded in the manifest.

19. The non-transitory machine-readable medium of claim 18, comprising instructions that when executed cause the processor to: In response to determining that a match is found between the first fingerprint and a fingerprint stored in a second container index: updating a reference count of the first data unit in the second container index; and Identifiers of the first data unit and the second container index are recorded in the manifest.

20. The non-transitory machine-readable medium of claim 15, comprising instructions that when executed cause the processor to: After updating the manifest to record the determined plurality of immutable storage locations, a new manifest is initialized.

Images