Storage location metadata for a deduplicated storage system

Abstract

Storage location metadata for a deduplication storage system is disclosed. Example implementations relate to deduplication operations within the storage system. Examples include: receiving a stream of data units to be stored in persistent storage of the deduplication storage system; generating an inventory to record the order in which the data units were received; indexing the data units into container indexes, wherein each container index records the fingerprint and immutable storage location of one or more data units; determining the immutable storage location recorded in the container indexes in response to determining that the inventory has reached a threshold size; and updating the inventory to record the determined immutable storage location.

Description

Background Technology

[0001] Data reduction techniques can be applied to reduce the amount of data stored in a storage system. Example data reduction techniques include deduplication. Deduplication identifies duplicate data units and attempts to reduce or eliminate the number of instances of duplicate data units stored in the storage system. Attached Figure Description

[0002] Some implementation methods are described with reference to the following figures.

[0003] Figure 1 This is a schematic diagram of an example storage system according to some implementation methods.

[0004] Figures 2A to 2B This is a diagram of an example data structure based on some implementation methods.

[0005] Figure 3 This is a diagram illustrating an example process based on some implementation methods.

[0006] Figure 4 This is a diagram illustrating an example process based on some implementation methods.

[0007] Figure 5 This is a schematic diagram of an example computing device according to some implementation methods.

[0008] Figure 6 This is a diagram illustrating an example process based on some implementation methods.

[0009] Figure 7 This is a diagram of an example machine-readable medium storing instructions according to some implementation methods.

[0010] Throughout the accompanying drawings, the same reference numerals denote similar but not necessarily identical elements. The drawings are not necessarily drawn to scale, and the size of some parts may be enlarged to illustrate the examples more clearly. Furthermore, the drawings provide examples and / or embodiments consistent with the description; however, the description is not limited to the examples and / or embodiments provided in the drawings. Detailed Implementation

[0011] In this disclosure, unless the context clearly indicates otherwise, the terms “a,” “an,” or “the” are intended to include the plural form as well. Similarly, when used in this disclosure, the terms “includes / including / comprises / comprising” or “have / having” indicate the presence of the element but do not preclude the presence or addition of other elements.

[0012] In some examples, a storage system can back up a collection of data (referred to herein as a “stream” or “data stream”) in deduplicated form, thereby reducing the amount of storage space required to store the data stream. The storage system can create “backup items” to represent the data stream in deduplicated form. The storage system can perform a deduplication process, including breaking down the data stream into discrete data units (or “chunks”) and determining the “fingerprints” of these incoming data units (described below). Further, the storage system can compare the fingerprints of the incoming data units with the fingerprints of stored data units, and can thereby determine which incoming data units are duplicates of previously stored data units (e.g., when the comparison indicates a matching fingerprint). In the case of a duplicate data unit, the storage system can store a reference to the previously stored data unit instead of storing the duplicate incoming data unit. The process of receiving and deduplicating inbound data streams can be referred to herein as the storage system’s “data ingestion” process.

[0013] As used herein, a "fingerprint" refers to the value obtained by applying a function to the contents of a data unit (where "contents" can include all or a subset of the contents of the data unit). Examples of functions that can be applied include hash functions that generate a hash value based on the contents of an input data unit. Examples of hash functions include cryptographic hash functions, such as Secure Hash Algorithm 2 (SHA-2) hash functions (e.g., SHA-224, SHA-256, SHA-384, etc.). In other examples, other types of hash functions or other types of fingerprint functions may be used.

[0014] A “storage system” may include storage devices or arrays of storage devices. A storage system may also include multiple storage controllers that manage access to the storage devices(s). A “data unit” may refer to any portion of data in the storage system that can be individually identified. In some cases, a data unit may refer to a block, a collection of blocks, or any other portion of data. In some examples, the storage system may store data units in persistent storage devices. Persistent storage devices may be implemented using one or more persistent (e.g., non-volatile) storage devices (such as multiple disk-based storage devices (e.g., multiple hard disk drives (HDDs)), multiple solid-state drives (SSDs) (such as multiple flash storage devices), etc., or combinations thereof). A “controller” may refer to hardware processing circuitry, which may include any or a combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, a digital signal processor, or other hardware processing circuitry. Alternatively, a “controller” may refer to a combination of hardware processing circuitry and machine-readable instructions (software and / or firmware) executable on the hardware processing circuitry.

[0015] In some examples, deduplication storage systems can use metadata to process inbound data streams (e.g., backup items). For example, such metadata may include a data recipe (also referred to herein as a “manifest”) specifying the receiving order of particular data units for each backup item. Further, such metadata may include item metadata to represent each received backup item (e.g., a data stream) in deduplicated form. Item metadata may include identifiers for a set of manifests and may indicate the order of these manifests. Processing each backup item may be referred to herein as a “backup process.” Subsequently, in response to a read request, the deduplication system can use the item metadata and the set of manifests to determine the receiving order of the data units, thereby recreating the original data stream of the backup item. Accordingly, the set of manifests may be a representation of the original backup item. The manifest may include a series of records, each representing a specific set(s) of data units(s). Records in the manifest may include one or more fields identifying container indices that index the data units (e.g., including storage information for the data units). For example, a container index may include one or more fields specifying location information of the stored data units (e.g., container, offset, etc.), compression and / or encryption characteristics of the stored data units, etc. Furthermore, the container index may include reference counts, which indicate the number of lists that reference each data unit.

[0016] In some examples, upon receiving a data unit (e.g., from a data stream), it can be matched against one or more container indices to determine if the same data unit is already stored in a container of the deduplication storage system. For example, the deduplication storage system can compare the fingerprint of the received data unit with fingerprints in one or more container indices. If no matching fingerprint is found in the searched container indices, the received data unit can be added to the container, and an entry for the received data unit can be added to the container index corresponding to that container. However, if a matching fingerprint is found in the searched container indices, it can be determined that the same data unit as the received data unit is already stored in the container. In response to this determination, the reference count of the corresponding entry is incremented, and the received data unit is not stored in the container (because it already exists in one of the containers), thus avoiding the storage of duplicate data units in the deduplication storage system. As used herein, the term "matching operation" can refer to the operation used to compare the fingerprints of a set of multiple data units (e.g., from a particular backup data stream) with fingerprints stored in container indices.

[0017] In some examples, deduplication storage systems may store data units and associated metadata objects in remote storage devices (e.g., a "cloud" or network storage service) instead of in a local file system. In such examples, processing a read request for a stored data unit may include: loading an appropriate inventory from the remote storage device into memory; reading the inventory to identify the appropriate container index; loading the container index from the remote storage device into memory; reading the container index to determine the location of the data unit stored in the remote storage device; and then loading the data unit from its location in the remote storage device into memory. Thus, performing a read request may include input / output (I / O) operations for loading the inventory, container index, and data unit. Therefore, completing a read request may consume network bandwidth and impact the processing performance of the deduplication storage system.

[0018] According to some embodiments of this disclosure, the controller of a deduplication storage system may assign immutable storage locations to data units during initial ingestion (e.g., when first received and stored) and may record the immutable storage locations in one or more container indices loaded in memory. As used herein, an “immutable storage location” for a data unit may refer to a storage address that does not change after being assigned to the data unit. The controller may record the order in which data units are received in a list loaded in memory. In some embodiments, when the information recorded in the list reaches a threshold level (e.g., a maximum level), the controller may iteratively traverse (multiple) container indices to extract the immutable storage locations of the data units. When the data ingestion process is complete (e.g., a backup process is complete), the controller may load the container indices and list from memory into persistent storage and may delete these items from memory. Subsequently, to access the stored data unit (e.g., in response to a read request), the controller may load the appropriate list from persistent storage into memory. The controller may read the list to determine the immutable storage location of the data unit and may then use the immutable storage location to access the data unit. The controller can perform read operations without having to load the container index from the storage device into memory (i.e., determine the storage address of the accessed data unit). In this way, the number of data transfers from the storage device to the memory can be reduced. Therefore, some implementations can improve the performance of a deduplication storage system. See below for reference. Figures 1 to 7 Further discussion of various aspects of the disclosed technology.

[0019] Figure 1 -Example Storage System

[0020] Figure 1An example system 105 is illustrated, including a storage system 100 and a remote storage device 190. According to some embodiments, the storage system 100 may include a storage controller 110, a memory 115, and a persistent storage device 140. The storage system 100 may be coupled to the remote storage device 190 via a network connection. The remote storage device 190 may be a network-based persistent storage facility or service (also referred to herein as a "cloud-based storage device"). In some examples, using the remote storage device 190 may incur financial costs based on the number of individual transfers.

[0021] Persistent storage device 140 may include one or more non-transitory storage media, such as hard disk drives (HDDs), solid-state drives (SSDs), optical disks, or combinations thereof. Memory 115 may be implemented using semiconductor memory such as random access memory (RAM). In some examples, storage controller 110 may be implemented via hardware (e.g., electronic circuitry) or a combination of hardware and programming (e.g., including at least one processor and instructions executable by said at least one processor and stored on at least one machine-readable storage medium). In some embodiments, memory 115 may include list 150 and container index 160. Further, persistent storage device 140 may store list 150 and container index 160. Remote storage device 190 may persistently store Container Entity Group (CEG) objects 170. Each CEG object 170 may be a container data structure configured to store multiple data units.

[0022] In some implementations, storage system 100 may perform deduplication on stored data. For example, storage controller 110 may divide an input data stream into data units and may include at least one copy of each data unit in at least one CEG object 170. Storage controller 110 may generate a list 150 to record the order in which data units in the data stream were received. List 150 may include pointers or other information indicating a container index 160 associated with each data unit. In some implementations, container index 160 may include a fingerprint (e.g., a hash) of the stored data unit used in the matching process of the deduplication procedure. Container index 160 may also include a reference count (e.g., an indication of the number of lists 150 referencing each data unit) of the data unit used in housekeeping processes (e.g., for determining whether to delete a stored data unit). References below Figures 2A to 2B Further discussion of example implementations of Listing 150, Container Index 160, and CEG Object 170.

[0023] In some implementations, container index 160 may include address information 155 for use in the reconstruction of deduplicated data, which records or indicates the storage location of data units. Further, in some implementations, listing 150 may include address information 155 for each data unit referenced by listing 150. For example, address information 155 (e.g., stored in container index 160 and listing 150) may include an immutable address identifying the CEG object 170 storing the data unit and the location of the stored data unit (within CEG object 170). References below... Figure 3 The example process for storing address information 155 in Listing 150 is discussed.

[0024] In some implementations, storage controller 110 may receive a read request to access stored data, and in response may access list 150 to determine the sequence of data units constituting the original data. Further, storage controller 110 may use address information 155 included in list 150 to determine the location of the stored data units (e.g., for each data unit, the corresponding CEG object 170, offset, etc.). In this way, storage controller 110 can determine the storage location of the data units without loading container index 160 into memory 115. Storage controller 110 can then read the data units from the determined location (e.g., by transferring the data units from remote storage device 190 to memory 115). References below... Figure 4 This section discusses an example procedure for performing a read request using address information 155 from Listing 150.

[0025] Note that, although Figure 1 An example is shown, but the implementation is not limited in this respect. For example, it is conceivable that some or all of the items in Listing 150 and Container Index 160 may be stored in remote storage device 190. In another example, it is conceivable that some or all of the CEG objects 170 may be stored in persistent storage device 140. In yet another example, it is conceivable that memory 115, persistent storage device 140, and / or remote storage device 190 may include other data objects or metadata. Furthermore, it is conceivable that storage system 100 may include additional devices and / or components, fewer components, different components, different arrangements, etc.

[0026] Figures 2A to 2B -Example Data Structure

[0027] Figure 2AA diagram illustrating an example data structure 200 used in deduplication according to some embodiments is shown. As shown, data structure 200 may include item metadata 202, a list 203, a container index 220, and a container entity group (“CEG”) object 250. In some examples, list 203, container index 220, and CEG object 250 may substantially correspond to list 150, container index 160, and CEG object 170, respectively (e.g., ...). Figure 1 Example implementation shown). Further, in some examples, the data structure 200 may be provided by the storage controller 110 (e.g., Figure 1 (As shown) Generate and / or manage.

[0028] In some implementations, project metadata 202 may include multiple manifest identifiers 205. Each manifest identifier 205 may identify a different manifest 203. In some implementations, the manifest identifiers 205 may be arranged in streaming order (i.e., based on the order in which the data units represented by the identified manifest 203 are received). Further, although for simplicity Figure 2A The illustrations show only one of each data structure, but data structure 200 may include multiple instances of project metadata 202, each instance including or pointing to one or more listings 203. In such an example, data structure 200 may include multiple listings 203. Listings 203 may include multiple listing records 210 referencing multiple container indices 220. Each container index 220 may include multiple cell metadata 225. Each instance of cell metadata 225 may index one or more data cells 260. Each CEG object 250 may include multiple data cells 260. Further, in some examples, CEG object 250 may include one or more groups or "entities" 255, where each entity 255 includes multiple data cells 260.

[0029] Now for reference Figure 2BThe diagram illustrates container index metadata 270 and inventory metadata 280. In some embodiments, inventory metadata 280 may be included in inventory 203 (e.g., in inventory record 210). Further, container index metadata 270 may be included in container index 220 (e.g., in cell metadata 225). In some embodiments, container index metadata 270 and inventory metadata 280 may each include cell address, cell length, and compression information. The cell address may be information stored in a field (or a combination of fields) that definitively identifies the storage location of one or more data cells. Further, the cell length may specify the data length of the data cell(s) stored at the cell address. In some embodiments, the cell address of a data cell may be recorded as three values ​​(e.g., stored in three fields) that identify a specific CEG object 250, a specific entity 255, and an offset within the specific entity 255, respectively. In other embodiments, the cell address may be an indication of being added to a deduplication storage system (e.g., Figure 1 The numerical value (referred to as "arrival number") of the arrival order (also referred to as "ingestion order") of data units in the system 105 shown. In some implementations, the cell address in the manifest metadata 280 does not change after being populated into the manifest metadata 280, and therefore can represent the immutable address of the data unit.

[0030] In some implementations, container index metadata 270 and / or manifest metadata 280 may use a run-length reference format to represent a range of consecutive data units (e.g., a portion of a data stream) stored within a single CEG object 250 (or a single entity 255). For example, a cell address field may record the offset of the start of the first data unit in the represented data range (within CEG object 250), and a cell length field may indicate the length of the represented data range. In another example, a cell address field may record the arrival number of the first data unit in the represented data range, and a cell length field may indicate the number N (where "N" is an integer) of data units in the data range following the first data unit specified by the arrival number in the cell address field. Data units in a data range may have consecutive arrival numbers (e.g., because they are consecutive in the ingested data stream). Thus, a data range may be represented by the arrival number of the first data unit in the data range (e.g., recorded in the cell address field) and the number N of the remaining data units in the data range (e.g., recorded in the cell length field). Additional data units within a data unit range that follow the first data unit can be determined qualitatively by calculating N arrival numbers that sequentially follow the specified arrival number of the first data unit, where these N arrival numbers identify the additional data units within the data unit range. For example, manifest metadata 280 may include an arrival number “X” in the cell address field and a quantity N in the cell length field to indicate a data unit range including the first data unit specified by arrival number X and subsequent data units specified by arrival number X+i, where i = 1 to i = N (inclusive) (where “i” is an integer). In this way, a run-length reference format can be used to identify all data units within a data unit range.

[0031] In some implementations, compression information may indicate how to compress or decompress the stored data units (whether compression was used, the type of compression code, the type of decompression code, the size of the decompression, checksum values, etc.). In some examples, during a read operation, compression information may be used to decompress the requested data unit (or a specific entity 255 that includes the requested data unit).

[0032] In some implementations, container index metadata 270 may include fingerprints and reference counts. A fingerprint may be a value obtained by applying a function (e.g., a hash function) to all or part of the contents of a data unit. A reference count may indicate the total number of manifest records 210 referencing the data unit. Further, in some implementations, fingerprints and reference counts may not be included in manifest metadata 280.

[0033] Note that, although Figures 2A to 2B An example of data structure 200 is shown, but the implementation is not limited in this respect. For example, it is conceivable that project metadata 202, list 203, container index 220, and CEG object 250 may include additional fields or elements, additional data structures, etc. In another example, it is conceivable that container index metadata 270 and / or list metadata 280 may include additional fields or elements.

[0034] Figure 3 - Example process for generating metadata

[0035] Figure 3 An example process 300 for generating metadata for a deduplication storage system, according to some implementations, is shown. In some examples, a storage controller 110 (such as...) can be used. Figure 1 Process 300 is executed as shown. Process 300 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, semiconductor storage device, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 300 can be executed by a single processing thread. In other embodiments, process 300 can be executed in parallel by multiple processing threads (e.g., simultaneously using a work chart and performing multiple housekeeping tasks).

[0036] Box 310 may include receiving data units to be stored in persistent storage of the deduplication storage system. Box 315 may include generating a fingerprint for the received data units. Box 320 may include comparing the generated fingerprint with a fingerprint stored in a container index of the deduplication storage system. For example, refer to... Figure 1 Storage controller 110 receives a stream of data units (e.g., backup items) to be stored in system 105. Storage controller 110 generates a fingerprint for the received data units and compares the generated fingerprint with the fingerprint included in container index 160.

[0037] Decision box 325 may include determining whether a match is found between the generated fingerprint and the fingerprint included in the container index (CI). If it is determined at decision box 325 that no match is found (“No”), process 300 may continue at box 330, including assigning an immutable storage location to the data unit. Box 335 may include storing the data unit in the assigned immutable storage location. Box 340 may include recording the fingerprint of the data unit and the immutable storage location in the container index. Box 345 may include recording the identifiers of the data unit and the container index in a manifest. For example, refer to... Figure 1If no match is identified between the fingerprint of the generated received data unit and the fingerprint included in container index 160, storage controller 110 stores the data unit in a new CEG object 170 from the immutable storage location. Storage controller 110 adds a new entry for the data unit to container index 160. The new entry in container index 160 can store container index metadata 270 (such as...). Figure 2B As shown, the container index metadata includes the fingerprint, reference count, cell address, cell length, and compression information of the data cell.

[0038] However, if a match is found (“Yes”) at decision box 325, process 300 can continue at box 350, including updating the reference count of the data unit in the container index that indexes the data unit. After box 350, process 300 can continue at box 345, including recording the identifiers of the data unit and the container index in a manifest. For example, refer to... Figure 1 If a match is identified between the fingerprint of the generated received data unit and the fingerprint stored in a specific container index 160, the storage controller 110 determines that a duplicate of the data unit has already been stored in the existing CEG object 170. Accordingly, the storage controller 110 increments the reference count (corresponding to the data unit) in the specific container index 160. Further, the storage controller 110 records information identifying the data unit and the specific container index 160 in the current listing 150.

[0039] Decision box 360 may include determining whether the list is full (e.g., whether the maximum number of entries representing data units has been reached). If not ("No"), process 300 may return to box 310 (i.e., receiving another data unit to be stored in persistent storage). In another case, if it is determined at decision box 360 that the list is full ("Yes"), process 300 may continue at box 370, including reading each container index recorded in the list to determine the immutable storage location of the data unit recorded in the list. Box 380 may include recording the determined immutable storage location of the data unit in the list. For example, refer to Figure 1 When it is determined that the current list 150 is full, the storage controller 110 accesses one or more container indices 160 that index a group of data units recorded in the current list 150 (e.g., loaded in memory 115), and reads the container indices 160 to obtain address information 115 indicating the immutable storage location of that group of data units. The storage controller 115 then stores the address information 115 of that group of data units in the list 150 (e.g., in list metadata 280 that includes unit address, unit length, and compression information).

[0040] In some implementations, the address information 155 for a range of consecutive data units can be represented in a run-length reference format. For example, if the range of consecutive data units is stored unchanged within a single CEG object 250 (or a single entity 255), the storage controller 110 can record the address information 155 for that range using a cell address field (which stores the offset or arrival number of the first data unit in the range) and a cell length field (which stores the number of data units following the first data unit in the range). The address information 115 stored in Listing 150 can then be used to execute a read request without having to load the container index 160 into memory 115. References below... Figure 4 This section discusses an example procedure for performing a read request using the address information 155 stored in Listing 150.

[0041] Box 390 may include initializing a new manifest. After box 390, process 300 may return to box 310 (i.e., receive another data unit to be stored in persistent storage). Alternatively, process 300 may complete. For example, refer to... Figure 1 After storing the immutable storage locations of the identified set of data units in list 150, the storage controller 110 initiates a new list 150 to record information about subsequently received data units.

[0042] Figure 4 - Example procedure for executing a read request

[0043] Figure 4 An example procedure 400 for executing a read request in a deduplication storage system, according to some embodiments, is shown. In some examples, a storage controller 110 (such as...) can be used. Figure 1 Process 400 is executed as shown. Process 400 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). The machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, semiconductor storage device, or magnetic storage device. The machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. In some embodiments, process 400 can be executed by a single processing thread. In other embodiments, process 400 can be executed in parallel by multiple processing threads (e.g., simultaneously using a work chart and performing multiple housekeeping jobs).

[0044] Box 410 may include receiving a read request for a data unit stored in a deduplication storage system. Box 420 may include a list identifying the record data units. Box 430 may include loading the identified list into memory. For example, refer to... Figure 1Storage controller 110 can receive read requests to access data units stored in remote storage device 190 (e.g., as part of a stored backup project). Storage controller 110 identifies a list 150 (located in persistent storage device 140) referencing the requested data unit and loads list 150 into memory 115. Further, in some examples, a container index 160 indexing the requested data unit is not loaded into memory 115 in response to a read request.

[0045] Box 440 may include obtaining the immutable storage location of the data unit from a list loaded in memory. Box 450 may include reading the data unit according to the immutable storage location obtained from the list loaded in memory. After box 450, process 400 may be completed. For example, refer to... Figure 1 The storage controller 110 reads address information 155 stored in list 150 (which is loaded in memory 115) to determine the immutable storage location of the requested data unit. The storage controller 110 then reads the requested data unit from the immutable storage location (e.g., in CEG object 170 stored in remote storage device 190). In this way, the storage controller 110 executes the read request without having to load container index 160 from persistent storage device 140 (or remote storage device 190) into memory 115. Therefore, process 400 can reduce the number of input / output (I / O) transfers required to execute the read request, and thereby improve the performance of storage system 100.

[0046] Figure 5 -Example computing device

[0047] Figure 5 A schematic diagram of an example computing device 500 is shown. In some examples, computing device 500 may substantially correspond to storage system 100 (e.g., Figure 1 (As shown). As illustrated, computing device 500 may include some or all of the following: hardware processor 502, memory 504, and machine-readable storage device 505 including instructions 510-560. Machine-readable storage device 505 may be a non-transitory medium. Instructions 510-560 may be executed by hardware processor 502 or by a processing engine included in hardware processor 502.

[0048] Instruction 510 can be executed to receive multiple data units to be stored in a persistent storage device of a deduplication storage system. Instruction 520 can be executed to generate a list to record the receiving order of the multiple data units. For example, refer to... Figure 1The storage controller 110 receives a stream of data units (e.g., backup items) to be stored in the deduplication storage system 100 and records the ingestion order of the data units in the current list 150.

[0049] Instruction 530 can be executed to index multiple data units into multiple container indexes, where each container index records the fingerprint and immutable storage location of one or more data units. For example, refer to... Figure 1 Storage controller 110 generates a fingerprint for the received data unit and compares the generated fingerprint with the fingerprint included in container index 160. If no match is identified between the generated fingerprint of the received data unit and the fingerprint included in container index 160, storage controller 110 stores the data unit in the immutable storage location in a new CEG object 170. Further, storage controller 110 records the fingerprint identifying the data unit and the information of the immutable storage location in container index 160. In another case, if a match is identified between the generated fingerprint of the data unit and the fingerprint included in container index 160, storage controller 110 determines that a duplicate of the data unit has already been stored in the existing CEG object 170. Accordingly, storage controller 110 increments the reference count (corresponding to the data unit) in the specific container index 160. Further, storage controller 110 records the information identifying the data unit and the specific container index 160 in the current listing 150.

[0050] Instruction 540 can be executed to determine multiple immutable storage locations recorded in multiple container indexes in response to determining that the list has reached a threshold size. Instruction 550 can be executed to update the list to record the determined multiple immutable storage locations. For example, refer to... Figure 1 When it is determined that the current list 150 is full, the storage controller 110 accesses one or more container indices 160 that index a group of data units recorded in the current list 150 (e.g., loaded in memory 115), and reads the container indices 160 to obtain address information 115 indicating the immutable storage location of that group of data units. The storage controller 115 then stores the address information 115 of that group of data units in the list 150 (e.g., in list metadata 280 that includes unit address, unit length, and compression information).

[0051] Figure 6 - Example process for generating metadata

[0052] Figure 6 An example process 600 for generating metadata for a deduplication storage system, according to some implementations, is shown. In some examples, a storage controller 110 (such as...) can be used. Figure 1Process 600 is executed as shown. Process 600 can be implemented in hardware or a combination of hardware and programming (e.g., machine-readable instructions executable by (multiple) processors). Machine-readable instructions can be stored in a non-transitory computer-readable medium such as an optical storage device, a semiconductor storage device, or a magnetic storage device. Machine-readable instructions can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc.

[0053] Box 610 may include receiving multiple data units to be stored in the persistent storage device of the deduplication storage system by the storage controller of the deduplication storage system. Box 620 may include generating an inventory by the storage controller to record the receiving order of the multiple data units. Box 630 may include indexing the multiple data units into multiple container indexes by the storage controller, wherein each container index records the fingerprint and immutable storage location of one or more data units.

[0054] Box 640 may include the storage controller determining whether the inventory has reached a threshold size. Box 650 may include, in response to determining that the inventory has reached the threshold size, the storage controller determining multiple immutable storage locations recorded in multiple container indexes. Box 660 may include the storage controller updating the inventory to record the determined multiple immutable storage locations. Boxes 610-660 may generally correspond to the reference instructions 510-550 above (e.g., ...). Figure 5 The example described is shown below.

[0055] Figure 7 -Example machine-readable media

[0056] Figure 7 A machine-readable medium 700 storing instructions 710-750 according to some embodiments is shown. Instructions 710-750 can be executed by a single processor, multiple processors, a single processing engine, multiple processing engines, etc. The machine-readable medium 700 can be a non-transitory storage medium, such as an optical storage medium, a semiconductor storage medium, or a magnetic storage medium. Instructions 710-750 may generally correspond to the referenced instructions 510-550 above (e.g., ...). Figure 5 The example described is shown below.

[0057] Instruction 710 can be executed to receive multiple data units to be stored in a persistent storage device of a deduplication storage system. Instruction 720 can be executed to generate an inventory to record the receiving order of the multiple data units. Instruction 730 can be executed to index the multiple data units into multiple container indexes, wherein each container index records the fingerprint and immutable storage location of one or more data units.

[0058] Instruction 740 can be executed to determine multiple immutable storage locations recorded in multiple container indexes in response to determining that the list has reached a threshold size. Instruction 750 can be executed to update the list to record the determined multiple immutable storage locations.

[0059] According to some embodiments of this disclosure, the controller of a deduplication storage system can assign immutable storage locations to data units during initial ingestion and can record the immutable storage locations in one or more container indices loaded into memory. In some embodiments, when the information recorded in the inventory reaches a threshold level, the controller can iteratively traverse the container indices(s) to retrieve the immutable storage location of the data unit. Subsequently, in response to a read request, the controller can load the appropriate inventory from persistent storage into memory. The controller can read the inventory to determine the immutable storage location of the data unit and can then access the data unit using the immutable storage location. The controller can perform the read operation without having to load the container indices into memory. In this way, the number of data transfers from storage device to memory can be reduced. Therefore, some embodiments can improve the performance of the deduplication storage system.

[0060] Note that, although Figures 1 to 7 Various examples are shown, but the implementation methods are not limited in this respect. For example, see references... Figure 1 It is conceivable that the functionality of the aforementioned storage controller 110 can be included in any other engine or software of the storage system 100. Other combinations and / or variations are also possible.

[0061] Data and instructions are stored in a corresponding storage device implemented as one or more computer-readable or machine-readable storage media. Storage media include various forms of non-transitory memory, including: semiconductor memory devices such as dynamic random access memory or static random access memory (DRAM or SRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory; magnetic disks such as fixed floppy disks and removable disks; other magnetic media, including magnetic tape; optical media such as optical discs (CDs) or digital video discs (DVDs); or other types of storage devices.

[0062] Note that the instructions discussed above may be provided on a single computer-readable or machine-readable storage medium, or alternatively, on multiple computer-readable or machine-readable storage media distributed across a large system with potentially multiple nodes. Such one or more computer-readable or machine-readable storage media are considered part of an article (or artifact). An article or artifact can refer to any single or multiple manufactured components. The one or more storage media may be located in a machine that executes the machine-readable instructions, or at a remote site from which the machine-readable instructions can be downloaded for execution via a network.

[0063] In the foregoing description, numerous details have been set forth to facilitate understanding of the subject matter disclosed herein. However, embodiments may be practiced without some of these details. Other embodiments may include modifications and variations of the details discussed above. The appended claims are intended to cover such modifications and variations.

Claims

1. A computing device, comprising: processor; Memory; as well as A machine-readable storage device storing instructions that can be executed by the processor to perform the following operations: Receive multiple data units in a persistent storage device to be stored in a deduplication storage system; Generate a list to record the receiving order of the multiple data units; The plurality of data units are indexed into a plurality of container indices, wherein each container index records the fingerprint and immutable storage location of one or more data units; In response to determining that the list has reached a threshold size, multiple immutable storage locations recorded in the plurality of container indices are determined; Update the list to record the identified multiple immutable storage locations; In response to a read request for a specific data unit, the updated inventory is used to determine the specific immutable storage location of the specific data unit; and Read the specific data unit from the specific immutable storage location determined using the updated inventory.

2. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations after updating the inventory to record the determined plurality of immutable storage locations: receiving the read request for the particular data unit, wherein The specific data unit is included in the plurality of data units stored in the persistent storage device; The updated list is identified based on the read request for the specific data unit; The identified updated list is loaded into the memory; The specific immutable storage location of the specific data unit is determined using the identified updated list loaded in the memory; as well as The specific data unit is read from the specific immutable storage location determined using the identified updated inventory loaded in the memory.

3. The computing device of claim 2, wherein, The specific data unit is indexed by a specific container index, wherein the specific container index is not loaded into the memory to execute the read request for the specific data unit.

4. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: Generate a first fingerprint for a first data unit included in a plurality of received data units; The first fingerprint is compared with the fingerprints stored in the plurality of container indices; In response to determining that no match is found between the first fingerprint and the fingerprints stored in the plurality of container indices: Assign the first immutable storage location to the first data unit; The first data unit is stored in the first immutable storage location; Record the first fingerprint and the first immutable storage location in the first container index; as well as The identifiers of the first data unit and the first container index are recorded in the list.

5. The computing device of claim 4, further comprising instructions executable by the processor to perform the following operations: In response to determining that a match is found between the first fingerprint and the fingerprint stored in the second container index: Update the reference count of the first data unit in the second container index; and The identifiers of the first data unit and the second container index are recorded in the list.

6. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: After updating the list to record the identified multiple immutable storage locations, a new list is initialized.

7. The computing device of claim 1, further comprising instructions executable by the processor to perform the following operations: In response to determining that a range of consecutive data units is stored within a single data object, a run-length reference value is recorded in the inventory to indicate the immutable storage location of the range of consecutive data units within the single data object.

8. The computing device of claim 7, wherein, The individual data object is a Container Entity Group ("CEG") object stored on a cloud-based remote storage device.

9. A method comprising: The storage controller of the deduplication storage system receives multiple data units to be stored in the persistent storage device of the deduplication storage system; The storage controller generates a list to record the receiving order of the plurality of data units; The storage controller indexes the plurality of data units into a plurality of container indices, wherein each container index records the fingerprint and immutable storage location of one or more data units; The storage controller determines whether the list has reached the threshold size; In response to determining that the list has reached the threshold size, the storage controller determines multiple immutable storage locations recorded in the plurality of container indices; The storage controller updates the inventory to record the identified multiple immutable storage locations; In response to a read request for a specific data unit, the storage controller uses the updated inventory to determine the specific immutable storage location of the specific data unit; and The storage controller reads the specific data unit from the specific immutable storage location determined using the updated inventory.

10. The method of claim 9, further comprising, after updating the inventory to record the identified plurality of immutable storage locations: receiving the read request for the particular data unit, wherein, The specific data unit is included in the plurality of data units stored in the persistent storage device; The updated list is identified based on the read request for the specific data unit; The identified updated list is loaded into the memory; The specific immutable storage location of the specific data unit is determined using the identified updated list loaded in the memory; as well as The specific data unit is read from the specific immutable storage location determined using the identified updated inventory loaded in the memory.

11. The method of claim 10, wherein, The specific data unit is indexed by a specific container index, wherein the specific container index is not loaded into the memory to execute the read request for the specific data unit.

12. The method of claim 9, comprising: Generate a first fingerprint for a first data unit included in a plurality of received data units; The first fingerprint is matched with fingerprints stored in the plurality of container indices; In response to determining that no match is found between the first fingerprint and the fingerprints stored in the plurality of container indices: Assign the first immutable storage location to the first data unit; The first data unit is stored in the first immutable storage location; Record the first fingerprint and the first immutable storage location in the first container index; as well as The identifiers of the first data unit and the first container index are recorded in the list.

13. The method of claim 12, comprising: In response to determining that a match is found between the first fingerprint and the fingerprint stored in the second container index: Update the reference count of the first data unit in the second container index; as well as The identifiers of the first data unit and the second container index are recorded in the list.

14. The method of claim 9, comprising: After updating the list to record the identified multiple immutable storage locations, a new list is initialized.

15. A non-transitory machine-readable medium storing instructions that, when executed, cause a processor to perform the following operations: Receive multiple data units in a persistent storage device to be stored in a deduplication storage system; Generate a list to record the receiving order of the multiple data units; The plurality of data units are indexed into a plurality of container indices, wherein, Each container index records the fingerprint and immutable storage location of one or more data units; In response to determining that the list has reached a threshold size, multiple immutable storage locations recorded in the plurality of container indices are determined; Update the list to record the identified multiple immutable storage locations; In response to a read request for a specific data unit, the updated list is used to determine the specific immutable storage location of the specific data unit; as well as Read the specific data unit from the specific immutable storage location determined using the updated inventory.

16. The non-transitory machine-readable medium of claim 15, comprising instructions that, at execution time, cause the processor to perform the following operations after updating the inventory to record the identified plurality of immutable storage locations: receiving the read request for the particular data unit, wherein, The specific data unit is included in the plurality of data units stored in the persistent storage device; The updated list is identified based on the read request for the specific data unit; The identified updated list is loaded into the memory; The specific immutable storage location of the specific data unit is determined using the identified updated list loaded in the memory; as well as The specific data unit is read from the specific immutable storage location determined using the identified updated inventory loaded in the memory.

17. The non-transitory machine readable medium of claim 16, wherein, The specific data unit is indexed by a specific container index, wherein the specific container index is not loaded into the memory to execute the read request for the specific data unit.

18. The non-transitory machine-readable medium of claim 15, comprising instructions that, upon execution, cause the processor to perform the following operations: Generate a first fingerprint for a first data unit included in a plurality of received data units; The first fingerprint is compared with the fingerprints stored in the plurality of container indices; In response to determining that no match is found between the first fingerprint and the fingerprints stored in the plurality of container indices: Assign the first immutable storage location to the first data unit; The first data unit is stored in the first immutable storage location; Record the first fingerprint and the first immutable storage location in the first container index; as well as The identifiers of the first data unit and the first container index are recorded in the list.

19. The non-transitory machine-readable medium of claim 18, comprising instructions that, upon execution, cause the processor to perform the following operations: In response to determining that a match is found between the first fingerprint and the fingerprint stored in the second container index: Update the reference count of the first data unit in the second container index; and The identifiers of the first data unit and the second container index are recorded in the list.

20. The non-transitory machine-readable medium of claim 15, comprising instructions that, upon execution, cause the processor to perform the following operations: After updating the list to record the identified multiple immutable storage locations, a new list is initialized.

Images