STORING A SMALL OBJECT REPRESENTATION IN A DEDUPLICATION SYSTEM
The deduplication system optimizes storage and access for both large and small objects by using a hierarchy for large objects and direct references for small objects, addressing inefficiencies in existing systems and improving memory usage and access speed.
Patent Information
- Application Number
- DE102021126985
- Authority / Receiving Office
- DE · DE
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-04-23
- Filing Date
- 2021-10-19
- Publication Date
- 2024-11-14
- Estimated Expiration
- 2041-10-19
AI Technical Summary
Existing deduplication systems inefficiently handle both large and small objects, as they require similar storage and access methods, leading to increased memory consumption and latency for small objects due to unnecessary intermediate-level data structures.
A deduplication system that differentiates between large and small objects by using a hierarchy of data structures for large objects and a direct reference for small objects, omitting intermediate-level data structures to optimize storage and access efficiency.
This approach reduces memory consumption and latency for small objects while maintaining efficient deduplication and data access, allowing flexible handling of varying object sizes and types.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
BACKGROUND
[0001] A client computer, such as a server or similar, may store data in a primary storage array and run workloads against the data stored in the primary storage array. In some examples, the data stored in the primary storage array may be backed up to a computer system separate from both the client computing device and the primary storage array for redundancy and data protection purposes. In some examples, this computer system may store data in a deduplicated form to store the data more compactly.
[0002] US 2019 / 0 121 705 A1 describes a system that accesses item metadata of a backup item backed up to a remote object storage system in response to an event in the deduplication system, wherein the item metadata of the backup item includes range information that specifies a range of identification values for sub-objects of the backup item stored in the remote object storage system.
[0003] US 2012 / 0 233 522 A1 describes a segmented object storage system that divides files into a number of object segments, each segment corresponding to a part of the object, and that stores each segment individually in the cloud storage system.
[0004] US 2017 / 0 270 134 A1 refers to deduplicating parts of a data block when generating a manifest hierarchy for the data block.
[0005] US 2018 / 0 218 005 A1 describes a deduplication module comprising deduplication software, a build diagram interface and a chunk store interface. SHORT DESCRIPTION
[0006] A method according to claims 1 to 7, an article according to claims 8 to 13 and a computing device according to claims 14 to 20 are disclosed. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] The following detailed description refers to the drawings in which: Fig. is a block diagram of an example deduplication system for storing a small object representation of an object; Fig. is a block diagram of an example computing device including instructions executable to store a small object representation of an object; Fig. is a flowchart of an example method including storing a small object representation of an object; Fig. is a block diagram of an example deduplication system in which a small object representation contains a manifest; and Fig. is a block diagram of an example deduplication system for deduplicating manifests. DETAILED DESCRIPTION
[0008] A client computing device, such as a server, a storage array, etc., may back up data by storing the data in a computing system capable of performing deduplication on the data to store the data in a deduplicated form that is more compact than a non-duplicated form. Such a computing system capable of performing such deduplication of data may be referred to herein as a deduplication system and may be implemented by a backup appliance. In the examples described herein, a deduplication system may perform a deduplication process on an object, which in the examples described herein is a collection of data. A deduplication system may receive objects as data streams for deduplicated storage.
[0009] The individual objects made available to a deduplication system for deduplicated storage can be defined by the client system that makes the objects available to the deduplication system for storage, so the provided objects can vary in size and content type. For example, an object can represent a single file, an entire file system (or part of it), one or more virtual volumes, or similar, and a given client can make objects of different sizes available to the deduplication system. However, it can be inefficient to store deduplicated representations of large and small objects in the same way. To solve these problems, the examples described here can use a large object representation for large objects in a deduplication system and a small object representation for small objects in the deduplication system.
[0010] Fig. is a block diagram of an example deduplication system 100 for storing a small object representation of an object. The deduplication system 100 may be implemented by a computing device, such as at least one storage array, a backup appliance, or the like. The deduplication system 100 may include at least one processing resource 110 and at least one machine-readable storage medium 120 containing (e.g., encoded with) at least instructions 122 that may be executed by the at least one processing resource 110 to implement functionality described herein related to instructions 122.Functionalities described herein that are performed (or can be performed by) the deduplication system 100 may be implemented by the processing resource(s) 110 that execute the instructions 122 of the deduplication system 100, and may also be described as functionalities of the instructions 122 (i.e., functionalities that are performed by the processing resource(s) 110 when they execute the instructions 122).
[0011] The deduplication system 100 may store deduplicated representations of objects using a variety of data structures. The plurality of data structures may include container data structures (e.g., files) intended to store fixed or variable-sized portions of the data content of the object(s). Container data structures may also be referred to herein as containers. In the example of Fig. The deduplication system 100 uses each of the containers 250, 252, 254, and 256 to store the respective chunks 260 of the objects stored in the deduplication system 100 via deduplicated representations. For example, container 250 may store at least chunks 261-264, container 252 may store at least chunks 261-269, container 254 may store at least chunks 206-209, and container 250 may store at least chunks 201-204. Although a relatively small number of chunks 260 are depicted in each container for clarity in the figures, each container may contain more or fewer chunks in the examples described herein.
[0012] The deduplication system 100 may store metadata for the deduplicated representations of objects in a hierarchy 290 of various types of data structures. The hierarchy 290 may include top-level data structures, bottom-level data structures, and intermediate-level data structures that are separate from and located between the top-level and bottom-level data structures in the hierarchy 290. In the example of Fig. the hierarchy 290 may include a top-level data structure 210, intermediate-level data structures 212, 214 and 216 and bottom-level data structures 220, 222, 224 and 226.
[0013] Each lowest-level data structure contains a corresponding instance of chunk metadata 150 for each chunk 260 in a corresponding container data structure. In the example of Fig. The lower-level data structures 220, 222, 224, and 226 each contain instances of chunk metadata 150 for chunks 260 of the corresponding container data structures 250, 252, 254, and 256. A lowest-level data structure that stores instance(s) of chunk metadata 150 may be referred to herein as a container index or container index data structure.
[0014] Each instance of chunk metadata 150 is associated with a corresponding chunk 260 stored in one of the container data structures and contains metadata associated with the corresponding chunk. The metadata may include a chunk signature (representing the contents of the associated chunk), a reference count for the associated chunk, and a location identifier indicating the storage location of the associated chunk (e.g., a location within the container data structure in which it is stored). Chunk metadata 150 may also include a data identifier (e.g., an arrival number) that identifies that instance of chunk metadata 150 itself.
[0015] In the example of Fig. the lowest-level data structure 220 corresponds to the container data structure 250 and contains a corresponding instance of chunk metadata 150 for each of the chunks 260 stored in the container data structure 250. For example, the lowest-level data structure 220 contains an instance of chunk metadata 150 (in Fig. shown) for chunk 261 of container 250. The chunk metadata 150 for chunk 261 includes a data identifier 230 (to identify this instance of chunk metadata 150 itself), a chunk signature SIG-1 for chunk 261, a reference number of 2 for chunk 261, and a location 80 that specifies a location of chunk 261 (e.g., an offset within the container data structure 250).
[0016] The also in Fig. The lowest-level data structure 222 shown, which is associated with the container 252, contains chunk metadata 150 for chunk 266 of the container 252. The chunk metadata 150 for chunk 266 includes the data identifier 242, the chunk signature SIG-5, a reference count of 3, and the storage location 82. The lowest-level data structure 224 associated with the container 254 contains the chunk metadata 150 for the chunk 206 of the container 254. The chunk metadata 150 for the chunk 206 includes the data identifier 236, the chunk signature SIG-3, a reference count of 1, and the location 84. Also in Fig. Shown are multiple instances of chunk metadata 150 contained in the bottom-level data structure 226 associated with the container 256, including: chunk metadata 150 for chunk 201 (including identifier 249, signature SIG-2, a reference number of 4, and location 85); chunk metadata 150 for chunk 203 (including identifier 270, signature SIG-4, a reference number of 2, and location 86); and chunk metadata 150 for chunk 204 (including identifier 272, signature SIG-7, a reference number of "7," and location 87). Although a relatively small number of instances of chunk metadata 150 in each lower-level data structure are shown in the figures for clarity, each lower-level data structure in the examples described here may contain more instances of chunk metadata 150.
[0017] Intermediate-level data structures of hierarchy 290, if present, are separate from and located between the top-level and bottom-level data structures in hierarchy 290. A top-level data structure 210 in hierarchy 290 may store one or more object identifiers (e.g., 20, 30), each associated with a corresponding object stored (in a deduplicated representation) in deduplication system 100. In some examples, a client system of deduplication system 100 may use the object identifier(s) to access the respective object(s) stored in deduplication system 100. A top-level data structure 210 may map each of the object identifiers stored therein to a corresponding metadata reference. An associated metadata reference may point to other metadata of a deduplicated representation of an object stored in the deduplication system 100 (e.g.,B. specify a location). In such examples, for each object identifier stored in top-level data structure 210, the object identifier is associated with a particular object, and top-level data structure 210 maps the object identifier to a metadata reference to other metadata of a deduplicated representation of the particular object.
[0018] In the example of Fig. The top-level data structure 210 (which may also be referred to as a catalog or catalog data structure 210) may store an object identifier (ID) 20 (e.g., a key) associated with a customer-provided object 10 stored in a deduplicated representation in the deduplication system 100. The top-level data structure 210 associates the object identifier 20 (associated with the object 10) with a reference 25, which is a reference to other metadata (intermediate-level data structure 212 in the example of Fig. ) of the deduplicated representation of object 10. Although, for clarity, a relatively small number of mappings between object identifiers and references in top-level data structure 210 are shown in the figures, each top-level data structure in the examples described herein may contain more or fewer such mappings.
[0019] As previously mentioned, the intermediate-level data structures of hierarchy 290, if any, are separate from and located between the top-level and bottom-level data structures of hierarchy 290. Each intermediate-level data structure may be referenced by (e.g., by pointing to) a top-level data structure or another (higher) intermediate-level data structure in hierarchy 290. Each intermediate-level data structure may point to (e.g., by pointing to) another (lower) intermediate-level data structure in hierarchy 290 or to one or more bottom-level data structures. In hierarchy 290, a first data structure may be considered higher than a second data structure to which the first data structure points or references. In hierarchy 290, a first data structure may be considered lower than a second data structure that points to or references the first data structure.
[0020] In the example of Fig. Top-level data structure 210 may reference intermediate-level data structure 212, which in turn may reference intermediate-level data structures 214 and 216. Intermediate-level data structure 214 may reference bottom-level data structures 220 and 222 (e.g., instances of chunk metadata 150 therein), and intermediate-level data structure 216 may reference bottom-level data structures 224 and 226 (e.g., instances of chunk metadata 150 therein). In some examples, a deduplicated representation of a customer-provided object may include one or more manifests that collectively specify an ordering of the chunks (stored in the deduplication system) that, when combined in that order, would form a reconstituted (or rehydrated) version of the customer-provided object stored in the deduplicated representation in the deduplication system.The order of the chunks specified in a manifest can be referred to here as the reconstruction order. In these examples, a manifest can represent an entire object or a part of an object (in which case, multiple manifests for multiple parts of an object together represent the entire object).
[0021] In the examples described here, an intermediate-level data structure may comprise a manifest for an object (or part of that object) stored in the deduplication system. The manifest for the object (or part of it) may represent the collection of chunks that comprise the object (or part of it) through references to chunk metadata associated with each of those chunks. A manifest for the object (or part of it) may represent a reconstruction order for the chunks that comprise the object (or part of it). For example, a manifest may contain the reconstruction order for the chunks by storing the references to the chunk metadata for those chunks in an order that represents the reconstruction order for the chunks (or in any other suitable manner).
[0022] In the example of Fig. The intermediate-level data structure 214 includes a manifest 160 with direct references 162 to the chunk metadata 150 in the bottom-level data structure(s) (e.g., 220, 222, etc.). The manifest 160 may store the references 162 to the chunk metadata 150 for a first portion of the object 10 associated with the large object ID 20. The manifest 160 may store the references 162 in a reconstruction order for the first portion of the object 10. The references 162 include a reference to chunk metadata 150 with ID 230 stored in the lowest level data structure 220 (“BL-220”) and a reference to chunk metadata 150 with ID 242 stored in the lowest level data structure 222 (“BL-222”).
[0023] The intermediate-level data structure 216 includes a manifest 164 with direct references 166 to chunk metadata 150 in bottom-level data structure(s) (e.g., 224, 226, etc.). The manifest 164 may store the references 166 to the chunk metadata 150 for a second portion of the object 10 associated with the large object ID 20. The manifest 164 may store the references 166 in a reconstruction order for the second portion of the object 10. References 166 include a reference to chunk metadata 150 with ID 236 stored in bottom-level data structure 224 ("BL-224") and a reference to chunk metadata 150 with ID 272 stored in bottom-level data structure 226 ("BL-226"). Since a manifest may represent a reconstruction order for at least part of an object, it may also be referred to herein as a partial index.Although a relatively small number of references are shown in the manifests shown in the figures for clarity, each manifest in the examples described here may contain more or fewer references.
[0024] In the examples described herein, a deduplicated representation of an object may include the data and metadata of the deduplication system 100 that may be used to reconstruct a complete (i.e., non-duplicated or rehydrated) version of the object. In the example of Fig. The deduplicated representation for object 10 may include top-level data structure 210, intermediate-level data structures 212, 214, and 216, and bottom-level data structures 220, 222, 224, and 226 (which form a tree-like structure for accessing each of the corresponding chunks 260 for object 10), as well as various chunks 260 stored in the containers of deduplication system 100 and referenced by the deduplicated representation metadata. In the examples described herein, data, metadata, and data structures of deduplication system 100 may be part of multiple different deduplicated representations of objects stored in deduplication system 100.
[0025] In such examples, to access portions of object 10 (e.g., for full reconstruction, selective access to data of object 10, etc.), deduplication system 100 may start with the large object ID 20 associated with object 10. Based on the mapping of large object ID 20 to reference 25 in data structure 210, deduplication system 100 may follow reference 25 to intermediate-level data structure 212. From there, deduplication system 100 may follow each of the references 168 from intermediate-level data structure 212 to data structures 214, 216, etc. Although for clarity a relatively small number of references 168 are illustrated in the intermediate-level data structure 212 shown in the figures, an intermediate-level data structure in the examples described herein may contain more or fewer such references.
[0026] The deduplication system 100 may use the respective manifests 160, 164, etc. contained in each of the intermediate-level data structures 214, 216, etc., to access the corresponding chunks 260 of the object 10 (and reconstruct them in the reconstruction order, if necessary). For example, using the manifest 160, the deduplication system 100 may use a first reference 162 to access the chunk metadata 150 with ID 230 in the bottom-level data structure 220. From this chunk metadata 150, the deduplication system 100 may access the chunk 261 at location 80 of the container 250. The deduplication system 100 may use a second reference 162 to access chunk metadata 150 with ID 242 in the bottom-level data structure 220 and, from that chunk metadata 150, access chunk 266 at location 82 of the container 252.In the same way, the deduplication system 100 can access each chunk referenced by the manifest 160 via each of the references 162.
[0027] The deduplication system 100 may similarly access chunks 260 based on additional manifests of intermediate-level data structures referenced by the intermediate-level data structure 212 (i.e., manifests for object 10). For example, the deduplication system 100 may use each of the references 166 of the manifest 164 to access instances of chunk metadata 150, including using a first reference 166 to access chunk metadata 150 with ID 236 in the bottom-level data structure 224 and accessing chunk 206 at location 84 of the container 254 from that chunk metadata 150. The deduplication system 100 may use a second reference 166 to access the chunk metadata 150 with ID 272 in the bottom-level data structure 226 and from that chunk metadata 150 to access the chunk 204 at location 85 of the container 256.
[0028] The hierarchy 290 of data structures, as described above, can be efficient for representing large objects (e.g., on the order of GB to TB or more). For example, the tree-like structure of the hierarchy 290 of data structures can make accessing a specific data element of a large object more efficient than would be the case with metadata organized in a flat or monolithic structure. For example, the hierarchy 290 can require much less processing overhead than a flatter metadata layout, because traversing the tree-like structure of the hierarchy 290 often requires less processing overhead than searching a flatter metadata layout.
[0029] The hierarchy 290 of data structures can also enable more efficient use of memory resources when accessing the data compared to a flatter metadata layout. For example, with a hierarchy 290 of data structures, a deduplication system 100 can limit the amount of metadata loaded into memory to perform a particular access. For example, the amount of metadata loaded into memory can be limited to the data structures in the hierarchy 290 that are relevant for locating the requested data, thereby avoiding the loading of other data structures in the hierarchy 290 (e.g., manifest data structures) that are not used for a particular access request.In contrast, a flat metadata organization, or even a flatter metadata organization that uses larger individual data structures, may consume more storage space and require larger amounts of input / output (I / O) and latency to pull more data into memory to access the requested data.
[0030] While a hierarchy of data structures can be advantageous for large objects, it can be quite inefficient for small objects. For example, the hierarchy of data structures described above can be inefficient if the object size is close to a minimum block size that can be used by the deduplication system (e.g., due to a file system limit, a storage device limit, or similar). For an object that is the size of a single block (e.g., 4 KB), the use of two levels of intermediate-level data structures in a hierarchy 290 can take up twice as much space as the actual content of the represented data (i.e., 4 KB in this example). While the object represented in this example is 4 KB in size, each of the two intermediate levels of metadata for the object can take up at least another 4 KB, thus taking up twice as much space (e.g.,2 x 4KB = 8KB) as the actual content of the represented data (i.e., 4KB in this example). Similarly, accessing parts of the small object in such an example may mean that each of the intermediate-level data structures used for that object must be accessed in memory to access the small object, increasing memory consumption and the latency for retrieving parts of the small object.
[0031] To address these problems, the examples described here may store a small object representation of an object based on determining that the object is smaller than a threshold size. The small object representation of the given object may include a direct reference from a top-level data structure to metadata of the small object in a bottom-level data structure of the small object representation, where the direct reference omits any intermediate-level data structures separate from the top-level and bottom-level data structures of the small object representation.
[0032] In this way, the examples described here may be able to store small objects in a deduplication system while omitting intermediate-level data structures that, if used, could significantly increase the size of the deduplication representation of the small object (relative to the small size of the object itself) and also increase the memory consumption and latency when accessing the deduplication representation of the small object.Storing a small object via a small object representation, as described here, can also enable the small object to be stored in a deduplication system efficiently and in a way that allows its data to become part of a deduplication domain, so that it can be deduplicated against other (previously stored) data, so that other (later stored) data can be deduplicated against the small object's data, or both.
[0033] The examples described here allow for flexibility in the deduplication system, allowing large and small objects to be treated differently so that both can be handled efficiently. Such flexibility can be beneficial because a deduplication system cannot determine the size and contents of the object submitted to it for storage. As previously mentioned, the size and contents of objects presented to the deduplication system for storage can be determined by the client system presenting the objects for storage, and client systems can present small objects for storage in a deduplication system, particularly when the usage of backup, long-term, or secondary storage resources changes.
[0034] In the past, a deduplication system (e.g., a deduplicating backup appliance) was often used to store very large objects representing the data of an entire machine or file system. In such cases, the backed-up data was often left unused in the backup appliance until it was fully restored (if at all). However, this use of a backup appliance as a "cold storage tier" is complemented by a use of the backup appliances that is more like a primary storage tier. This can include models where workloads are run on backup appliances, such as launching a virtual machine on a backup appliance (with the data stored there) before being transferred to another system for further operations.Because backup appliances are treated more like a primary storage tier, there may be a greater need for random access to the data stored on the backup appliance, and therefore it may be desirable to reduce the amount of I / O and the latency of I / O for such random access requests. As described above, the examples described here can reduce I / O and the corresponding I / O latency for small objects by omitting the intermediate-level data structures (and thus the retrieval of these data structures into memory to interact with the small object data).
[0035] Furthermore, such changes in backup appliance usage can also change the types of objects that clients store in backup appliances. For example, there are several scenarios in which backup appliances receive small objects from client systems, and in which these small objects may be subject to frequent changes. For example, a client system may use a very small lock file for its operations and mount this small file as an independent object for backup. Metadata about a backup may also be stored by a client system as a separate file, and the client system may mount this file as an independent object for storage in the backup appliance.
[0036] Before further discussing small object representations, an exemplary deduplication process is described below with respect to the deduplication system 100 of Fig. described to provide additional context. The deduplication system 100 may have one or more deduplication domains, each of which is a particular collection of data with which a deployed object can be deduplicated. In the example of Fig. is a part of a single deduplication domain. Although for simplicity, abbreviated versions of the deduplication domain and the deduplication domain data, metadata, and data structures are shown in Fig. As shown, the deduplication system 100 may contain more or less of each type of data structure of the hierarchy 290, more or fewer container data structures, and more or fewer instances of each contained data type.
[0037] In a Fig. In the example illustrated, the deduplication system 100 may receive the contents of an object 10 (e.g., as a data stream) to store it in a deduplicated representation in the deduplication system 100. The deduplication system 100 may divide the data of the object 10 into a plurality of portions of the object 10 and perform aspects of the deduplication process (e.g., reconciliation, etc.) for each portion of the object 10 in portions. In other examples, the object 10 may be treated as a single portion containing the entire contents of the object 10 (e.g., depending on the size of the object 10).
[0038] The deduplication system 100 may process each part of the object 10 separately, including processes of dividing (or chunking) the part into fixed-length (e.g., 4 KB) or variable-length sections, referred to herein as chunks, performing a matching process to identify duplicate chunks of the part (i.e., chunks whose contents are identical to other chunks already stored in the deduplication domain), storing a (full or compressed) copy of each chunk that has not been identified as a duplicate of a previously stored chunk, and, for each duplicate chunk, storing a reference (e.g., a pointer) to a stored copy of the chunk without storing the duplicate chunk again. In this way, a deduplication process can often avoid storing duplicates of the same chunk in a deduplication system.In some examples, a reference to the stored copy of the chunk may be a reference to an instance of the chunk metadata 150 for the chunk (i.e., an indirect reference to the chunk). By performing this process for each portion of the object 10, the deduplication system 100 may store a deduplicated representation of the object 10.
[0039] The deduplication system 100 may perform the matching process for a portion after the portion has been divided into chunks. The matching process may include determining the respective chunk signatures for each of the chunks (i.e., one per chunk). Chunk signatures are indicated in the figures using reference symbols of the format "SIG-N," where "N" represents a number. In the examples described herein, a chunk signature may consist of data representative of the contents of a chunk, derived by applying a signature function to the chunk. In some examples, the signature function may be a hash function, and the chunk signature may be a hash (or hash value) of the chunk generated by applying the hash function to the chunk. Any suitable hash function may be used to generate the chunk signature.In other examples, any other suitable type of signature function may be used instead of a hash function to generate a chunk signature (e.g., a function to generate a suitable type of fingerprint).
[0040] In the example of Fig. In a matching process for a portion of the object 10, the deduplication system 100 may, for each chunk of the portion, compare the chunk signature of the chunk with the chunk signatures present in at least one of the bottom-level (or container index) data structures (e.g., 220, 222, 224, 226, etc.) of the deduplication domain to determine whether any chunk signature in the scanned bottom-level data structure(s) matches the chunk signature of the given chunk. If a chunk signature that matches the chunk signature of the given chunk is found in the scanned lowest-level data structure(s), the deduplication system 100 may determine that a match has been found for the given chunk and may not store the current copy (i.e., the duplicate copy) of the chunk in the deduplication domain.Instead, the deduplication system 100 may represent the chunk by adding a reference to the matching, previously stored chunk to a manifest for the current portion. This reference may include a reference to the instance of chunk metadata 150 associated with the matching, previously stored chunk.
[0041] For example, the deduplication system 100 may, in a reconciliation process during the ingest of the object 10 from Fig. determine a chunk signature of "SIG-1" for a given chunk, search selected bottom-level data structures for a matching chunk signature, and find the matching chunk signature "SIG-1" in an instance of the chunk metadata 150 of the bottom-level data structure 220, thereby determining that a match has been found for the given chunk (e.g., chunk 261). In this example, the deduplication system 100 may not add the given chunk to a container of the deduplication domain (since doing so would involve storing a duplicate of the chunk 261). Instead, the deduplication system 100 may add a reference 162 to the instance of the chunk metadata 150 with ID 230 associated with the chunk 261 to the manifest 160 of the data structure 214.
[0042] In cases where no chunk signature matching the chunk signature of a given chunk is found in the scanned low-level data structure(s), the deduplication system 100 may determine that no match was found for the given chunk and add the given chunk to a container for storage in the deduplication domain. The deduplication system 100 may also add an instance of the chunk metadata 150 for the given chunk to a low-level data structure corresponding to the container to which the given chunk is being added. In such examples, a reference to the instance of the chunk metadata 150 for the given chunk is added to a manifest for the portion of the object 10 being processed.
[0043] As an example, let us consider a point in time before chunk 204 and its associated metadata are stored in the Fig. deduplication domain shown, and assume that the deduplication system 100 performs a matching process for a particular chunk 204 during the ingest of object 10. In such an example, the deduplication system 100 may determine a "SIG-7" chunk signature for the given chunk 204, search one or more lowest-level (container index) data structures (e.g., 220, 222, 224, 226, etc.), and determine that no matching "SIG-7" chunk signature is present in them. In this way, the deduplication system 100 may determine that no matching chunk signature was found in the deduplication domain.Based on this determination, the deduplication system 100 may add the chunk 204 to the container data structure 256, add a corresponding instance of chunk metadata 150 with ID 272 to the bottom-level data structure 226 (corresponding to the container 256), and add a reference 166 to the chunk metadata 150 with ID 272 to the manifest 164. In such examples, the deduplication system 100 may use the chunk signatures stored in bottom-level data structures to determine whether a given chunk of an ingested portion is a duplicate of a chunk already stored in a container data structure of the current deduplication domain, and if so, avoid storing a duplicate copy of that chunk in the deduplication domain.
[0044] Examples are given below in relation to the Fig. and with continued reference to Fig. described. Fig. is a block diagram of an example computer system 102 that includes instructions 122 that may be executed to store a small object representation of an object. In the example of Fig. The example computer system 102 includes at least one processing resource 110 and at least one machine-readable storage medium 120 that stores instructions 122 for storing objects. The instructions 122 may include instructions 124 for storing an object using a small object representation. The computer system 102 may be implemented by at least one computing device and may include one or more servers, a storage array, a backup appliance, or the like (or a combination thereof). The computer system 102 may implement a deduplication system, such as the deduplication system 100 of Fig. . The processing resource(s) 110, the machine-readable storage medium 120 and the instructions 122 of Fig. are the same as those in Fig. illustrated components of the system 100. The functionalities of the instructions 124 may also be described here as functionalities of the instructions 122, since the instructions 122 include the instructions 124.
[0045] Fig. is a flowchart of an example method 300 including storing a small object representation of an object. Although the execution of the method 300 is described below with reference to the deduplication system 100 of Fig. and the computer system 102 of Fig. described, other computer systems suitable for carrying out the method 300 may also be used, and the implementation of the method 300 is not limited to such examples. Although the flowchart in Fig. While a method shows a specific order for the execution of certain functions, the method presented is not limited to this order. For example, the functions shown sequentially may be executed in a different order, they may be executed simultaneously or partially simultaneously, or a combination thereof.
[0046] With reference to Fig. At 305 of method 300, instructions 122 of computer system 102 (when executed by at least one processing resource 110) may store in computer system 102 a large object representation of a first object that is larger than a threshold size. At 310, instructions 124 may determine that a second object is smaller than the threshold size. At 315, based on the determination at 310, instructions 124 may store a small object representation of the second object in computer system 102. The threshold size may be any suitable size consistent with the examples described herein.For example, the threshold size may be 10 MB, so instructions 122 may determine that an object less than 10 MB is smaller than the threshold size (and may be treated as a small object and stored via a small object representation), and instructions 122 may determine that an object greater than or equal to 10 MB is greater than or equal to the threshold size (and may be treated as a large object and stored via a large object representation). Instructions 122 may compare the size of an object to the threshold size based on the size of the object without deduplication (e.g., before deduplication or the size the object would have if restored to a non-deduplicated or rehydrated form).In other examples, the threshold size may be greater than or less than 10 MB, may be an adjustable parameter of computer system 102, and an appropriate threshold size may be determined empirically. In such examples, the determination of whether an object is smaller (or greater than or equal to) the threshold size may be made directly by comparing a size of the object (without deduplication) to the threshold size. In other examples, the determination may be made indirectly, e.g., based on the size of a manifest for the object in a deduplicated representation (since the size of such a manifest for an object is related to the size of the object without deduplication).
[0047] Examples are given with reference to the Fig. described in more detail. The functionalities described below with respect to the instructions 122 apply to both the instructions 122 of the deduplication system 100 of Fig. as well as for the computer system 102 of Fig. (including the functionalities of instructions 124 contained in instructions 122). In some examples, computer system 102 may be Fig. the deduplication system 100 from Fig. implement.
[0048] With reference to the Fig. Instructions 122 may receive a first object 10 for storage in a deduplicated representation in deduplication system 100. Instructions 122 may receive object 10 as a data stream and process object 10 in its entirety or divide it into parts that are deduplicated separately (as described above). In the example of Fig. Instructions 122 may determine that the first object 10 is larger than a threshold and, based on this determination, store a large object representation of the first object 10 in the deduplication system 100 at 305 of the method 300. The representation of the large object 10 may include metadata organized in a hierarchy 290 of data structures including: a top-level data structure 210; bottom-level data structures 220, 222, 224, 226, etc.; and intermediate-level data structures 212, 214, 216, etc., separate from the top-level data structure 201 and the bottom-level data structures 220, 222, 224, 226, etc. in the hierarchy 290.
[0049] As described above, instructions 122 may chunk object 10 as part of a deduplication process for object 10 and perform the matching process described above, after which each of the lowest-level data structures (220, 222, 224, 226, etc.) of the large object representation of object 10 includes an instance of chunk metadata 150 containing a storage location (e.g., 80) for a corresponding chunk (e.g., 261) of object 10. Each instance of chunk metadata 150 of the large object representation of object 10 also includes an ID for the instance of chunk metadata 150 (e.g., ID 230), a chunk signature (e.g., SIG-1) for a corresponding chunk (e.g., chunk 261) of object 10, and a reference number (e.g., 2) for the corresponding chunk of object 10 (each as described above with respect to Fig. described).
[0050] In the example of Fig. The large object representation of object 10 contains intermediate-level data structures (212, 214, 216, etc.) that contain a manifest representing a reconstruction order for the chunks of object 10. The reconstruction order can be maintained in the intermediate-level data structures via references to instances of chunk metadata 150 of the bottom-level data structures 220, 222, 224, 226, etc. of the large object representation of object 10.
[0051] In the examples described herein, a plurality of intermediate-level data structures may collectively contain a manifest, either by one of the intermediate-level data structures containing the entire manifest, or by more than one of the intermediate-level data structures each containing a manifest for a specific portion of an object, such that the manifests for the portions collectively form a manifest for the object as a whole. Furthermore, some intermediate-level data structures may not store any portion of a manifest. For example, intermediate-level data structures 214, 216, etc., may each contain a manifest 160, 164, etc., for a corresponding portion of object 10, such that the manifests for the corresponding portions of object 10 collectively form a manifest for object 10. Furthermore, although intermediate-level data structure 212 is part of the plurality of intermediate-level data structures 212, 214, 216, etc.that comprise a manifest for object 10, but the intermediate-level data structure 212 does not contain a manifest part because it is a higher-level data structure in the hierarchy 290 that lies above and references the intermediate-level data structures that contain the manifests of the parts of object 10.
[0052] In the examples described here, the intermediate-level data structures are separated from the top-level and bottom-level data structures and are located between them. Fig. For example, the intermediate-level data structures 212, 214, and 216 are separated from the top-level structure 210 and the bottom-level data structures 220, 222, 224, 226, etc., and lie between them in the hierarchy 290. In the examples described here, an intermediate-level data structure is located between a top-level data structure and a bottom-level data structure if, for example, a sequence of references from the top-level data structure to the bottom-level data structure passes through the intermediate-level data structure.For example, a sequence of references from top-level data structure 210 to bottom-level data structure 220 traverses intermediate-level data structure 212 to bottom-level data structure 220, including the following references: a reference 25 from top-level data structure 210 to intermediate-level data structure 212, a reference 168 from intermediate-level data structure 212 to intermediate-level data structure 214, and a reference 162 from intermediate-level data structure 214 to bottom-level data structure 220 (specifically, to the instance of chunk metadata 150 with ID 230). In this example, intermediate-level data structure 214 is also located between top-level data structure 210 and bottom-level data structure 220.
[0053] Another way in which an intermediate-level data structure, if present, is considered to be between a top-level data structure and a bottom-level data structure is if the top-level data structure is above the intermediate-level data structure in the hierarchy, and the bottom-level data structure is below the intermediate-level data structure in the hierarchy. For example, top-level data structure 210 is above intermediate-level data structure 212 (and intermediate-level data structures 214 and 216) in hierarchy 290, and the bottom-level data structure is below intermediate-level data structure 212 (and intermediate-level data structures 214 and 216) in hierarchy 290.
[0054] In the examples described herein, the intermediate-level data structures are separated from the top-level data structures and the bottom-level data structures (which are also separate from each other). In such examples, the deduplication system may allocate separate units of storage space for each top-level data structure, each intermediate-level data structure, and each bottom-level data structure. For example, a computer system such as deduplication system 100 may employ a storage allocation strategy with a smallest allocation (or block) size (e.g., 4 KB or similar). In such examples, the creation of any top-level, intermediate-level, or bottom-level data structure may include instructions 122 that allocate storage space with at least the smallest allocation size for the data structure. The storage space may initially be in volatile storage, such as RAM, or in non-volatile memory, or the like.allocated. Each such data structure may remain within the originally allocated storage space or grow beyond that initial allocation size. In such examples, each top-level, intermediate-level, and bottom-level data structure may be managed as a separate storage unit that may be managed separately from other such data structures. For example, each top-level, intermediate-level, and bottom-level data structure may be managed independently within the deduplication system 100, e.g., by being independently flushed from memory to persistent storage or independently reclaimed from persistent storage to memory.
[0055] Again referring to the Fig. , instructions 122 may receive a second object 40 for storage in a deduplicated representation in the deduplication system 100. The instructions 122 may receive the object 40 as a data stream and process the object 40 in its entirety or divide it into pieces to be deduplicated separately (as described above). In such examples, the instructions 122 may determine at 310 of the method 300 that the object 40 is smaller than the threshold size. Based on this determination, the instructions 122 may store a small object representation of the object 40 in the deduplication system 100 at 315 of the method 300.
[0056] The small object representation of the second object 40 may include a direct reference 35 from the top-level data structure 210 to the small object metadata in the bottom-level data structure 226 of the small object representation. The direct reference 35 is direct in that it omits any intermediate-level data structures that are separate from the top-level data structure 210 (containing the direct reference 35) and the bottom-level data structure 226 of the small object representation to which the direct reference 35 refers (e.g., points). The bottom-level data structure 226 of the small object representation contains chunk metadata 150 for each chunk of the second object 40, including one or more storage locations (e.g., location 87) for each chunk (e.g., chunk 204) of the second object 40.
[0057] In the examples described here, small object metadata in a bottom-level data structure can contain an instance of chunk metadata 150 (as in the example of Fig. ) or a manifest stored in the bottom-level data structure (as in the examples of Fig. , which are described below). In the example of Fig. The contents of the second object 40 may be contained in a single chunk 204 of the container 256, and the chunk 204 may have existed prior to the ingest of the object 40 or may have been added during the ingest of the object 40. In such examples, the small object metadata of the small object representation includes the instance of chunk metadata 150 with ID 272 that is for the single chunk 204 of the second object 40. In such examples, the direct reference 35 comprises a reference stored in the top-level data structure 210 to the instance of chunk metadata 150 with ID 272 that is for the single chunk 204 of the second object 40. This instance of chunk metadata 150 is stored in the bottom-level data structure 226 of the small object representation.In such examples, the direct reference 35 comprises a reference stored in the top-level data structure 210 and mapped to a small object identifier 30 associated with the second object 40.
[0058] In the example of Fig. During the reading of the object 40, the instructions 122 may determine that the contents of the second object 40 are small enough to be represented by a single chunk in the deduplication system 100. For example, the second object 40 may be no larger than a chunk size (or a maximum chunk size) used by the deduplication system (e.g., 4 KB). Based on the determination that the chunk 40 is small enough, the instructions 122 may store in the top-level data structure 210 a reference 35 to the instance of the chunk metadata 150 for the single chunk of the second object 40 (i.e., the instance of the chunk metadata 150 with ID 272, which is for the chunk 204).
[0059] Further examples of small object representations are given here in relation to the Fig. described. Fig. is a block diagram of an exemplary deduplication system 100 in which a small object representation contains a manifest. The deduplication system 100 of Fig. is the same as the deduplication system 100 from Fig. , although the example of Fig. some other functionalities of the deduplication system 100 than the example of Fig. shows.
[0060] As above with regard to Fig. described, a small object representation of a second object 40 may include a direct reference from the top-level data structure 210 to metadata of the small object in the bottom-level data structure 226 of the small object representation, and the metadata of the small object may be a manifest stored in the bottom-level data structure 226. In the example of Fig. A small object representation of a second object 40 may contain a direct reference 37 from the top-level data structure 210 to a manifest 211 stored in the bottom-level data structure 226. In some examples, the metadata of the small object may initially be an instance of the chunk metadata 150 (as in the example of Fig. ) and then move on to a manifesto 211, as in Fig. In other examples, the metadata of the small object may initially be a manifest such as manifest 211, as in Fig. shown.
[0061] An example where the deduplication system 100 converts the small object metadata from an instance of chunk metadata to a manifest is described below with respect to the Fig. In such examples, the instructions 122 may first store in the deduplication system 100 a small object representation of the second object 40, as described above with respect to Fig. described (including a direct reference 35 to an instance of chunk metadata 150 with ID 272). In some examples, after storing this small object representation of the second object 40, the instructions 122 may receive an updated version of the object 40 (see Fig. ), which is larger than the original version of object 40 of the example of Fig. In such examples, instructions 122 may determine that additional content 41 of the updated version of object 40 should be stored in the deduplicated representation of second object 40 in deduplication system 100. In some examples, instructions 122 may determine that the content of the updated second object 40 with the additional content 41 is both too large to be stored in a chunk and smaller than the threshold size.
[0062] Based on this determination, the instructions 122 can store in the bottom-level data structure 226 of the small object representation a manifest 211 for the updated second object 40 with the additional content 41. In the example of Fig. the content of the updated second object 40 is contained in a plurality of chunks 260, including chunks 201, 203, 204, etc. of the container 256. The manifest 211 for the updated second object 40 represents, via references 168 to instances of chunk metadata 150, a reconstruction order of the plurality of chunks 260 containing the content of the updated second object 40, including the additional content 41. The manifest 211 includes direct references 168 to instances of chunk metadata 150 in the bottom-level data structure(s). In the example of Fig. the references 168 include a reference to chunk metadata 150 with ID 272 stored in the bottom-level data structure 226 (“BL-224”), a reference to chunk metadata 150 with ID 270 in the data structure 226, and a reference to chunk metadata 150 with ID 249 in the data structure 226. The chunk metadata 150 with ID 272 is associated with the chunk 203, the chunk metadata 150 with ID 270 with the chunk 204, and the chunk metadata 150 with ID 249 with the chunk 201 in the example of Fig. .
[0063] In such an example of transitioning the metadata of the small object from an instance of the chunk metadata 150 to a manifest, the instructions 122 may invoke the top-level data structure 210 (see Fig. ) to include a direct reference 37 to Manifest 211 (see Fig. ) as a direct reference of the small object representation. In such examples, the small object metadata includes the manifest 211 for the updated second object 40. In such examples, the updated direct reference 37 is a reference to the manifest 211 stored in the bottom-level data structure 226 of the small object representation for the updated object 40.
[0064] Although the chunks 260 for the updated second object 40 in the example of Fig. Chunks 260 of the container 256, in other examples, the chunks 260 for the updated second object 40 may originate from any of the containers of the deduplication system 100. Although chunks 201, 203, and 204 are described as example chunks 260 for the updated second object 40, in various examples, more or fewer chunks 260 may be used to represent the contents of the updated second object 40, and in other examples, the manifest 211 may contain more or fewer references 168 than those in Fig. shown. In the example of Fig. The illustrated references 168 are references to instances of chunk metadata 150 in the bottom-level data structure 226 (i.e., the bottom-level data structure including manifest 211). In other examples, the manifest 211 for the updated second object 40 may include references to instances of chunk metadata 150 from several different bottom-level data structures of the deduplication system 100, and the chunks 260 representing the contents of the updated second object 40 may include chunks 260 from any containers of the deduplication system 100 (that are in the same deduplication domain).
[0065] In the examples described here, where the metadata of the small object includes a manifest, instructions 122 may select a bottom-level data structure in which to store the manifest from the bottom-level data structures of the deduplication domain of the deduplication system. Instructions 122 may make the selection in various ways. For example, instructions 122 may select a bottom-level data structure to which the manifest has an affinity or to which the manifest has the greatest affinity among the bottom-level data structures of the deduplication domain, based on one or more suitable affinity measures.
[0066] For example, a measure of the affinity between a manifest and a particular bottom-level data structure may be the number of references in the manifest to instances of chunk metadata 150 in the given bottom-level data structure. For example, if all references 168 of the manifest 211 are references to chunk metadata 150 in the bottom-level data structure 226, the instructions 122 may determine that the manifest 211 has the greatest affinity to the bottom-level data structure 226 (see Fig. ). In other examples, if the references 168 of the manifest 211 refer to chunk metadata 150 in various bottom-level data structures of a deduplication domain of the deduplication system 100, the instructions 122 may determine that the manifest 211 has the greatest affinity to one of the bottom-level data structures 220, 222, 224, 226, etc., to which it has the most references 168 (i.e., references to instances of chunk metadata 150 in those bottom-level data structures). For example, if the manifest 211 includes references 168 to instances of chunk metadata 150 in multiple bottom-level data structures 220, 222, 224, 226, etc., and contains the most references 168 to instances of chunk metadata 150 in the bottom-level data structure 226, the instructions 122 may determine that the bottom-level data structure 226 has the greatest affinity to the manifest 211 and, based on that, select the bottom-level data structure 226 for placement of the manifest 211.In other examples, if the manifest 211 includes the most references 168 to instances of chunk metadata 150 in another bottom-level data structure (e.g., 224), the instructions 122 may determine that the other bottom-level data structure (e.g., 224) has the greatest affinity to the manifest 211 and, based on this, select that other bottom-level data structure (e.g., 224) for placement of the manifest 211. In some examples, the instructions 122 may consider only the unique references 168 in the manifest 211 to determine the affinity.
[0067] In some examples, instructions 122 may use other affinity measures and base the selection of a bottom-level data structure for placement of a manifest on those measures. For example, instructions 122 may use the Jaccard index to determine the similarity between a manifest and each bottom-level data structure in the deduplication domain (or between a manifest and a suitable candidate subset of the bottom-level data structures in the deduplication domain). For example, instructions 122 may determine that a manifest has the greatest affinity to a particular bottom-level data structure that has the greatest similarity according to the Jaccard index measure of similarity among a plurality of bottom-level data structures.In such examples, the Jaccard Index measure of similarity may be used, for example, by comparing a first sentence representing a given bottom-level data structure and a second sentence representing a given manifest, where the contents of the first sentence represent the instances of chunk metadata 150 contained in the given bottom-level data structure, and the contents of the second sentence represent the instances of chunk metadata 150 (in corresponding bottom-level data structures) referenced by references 168 in the given manifest.
[0068] In other examples, instructions 122 may use a maximum or minimum identification technique to determine affinity. For example, instructions 122 may determine a maximum chunk signature ("SIG-X") referenced by manifest 211 and determine that manifest 211 has an affinity to the bottom-level data structure with that maximum chunk signature. In such examples, instructions 122 may select, for placement of manifest 211, the bottom-level data structure for which manifest 211 has been determined to have affinity. In such examples, the affinity may be binary (i.e., has affinity or has no affinity) rather than a measure with additional gradations (e.g., greater or lesser affinity).For example, for each reference 168 (or each unique reference 168) in the manifest 211, instructions 122 may determine the SIG-X chunk signature in the instance of chunk metadata 150 referenced by the reference 168 and determine the largest chunk signature among the determined chunk signatures. Instructions 122 may then determine that the manifest 211 has an affinity to the bottom-level data structure that has this largest chunk signature in one of its instances of chunk metadata 150. For example, the manifest 211 may contain references 168 to instances of chunk metadata 150 in the bottom-level data structures 224 and 226, and the instructions 122 may determine that the chunk signature SIG-7 is the largest chunk signature among those referenced by the references 168 (i.e.indirectly), and based on this determination, instructions 122 may determine that manifest 211 has an affinity to bottom-level data structure 226 having the SIG-7 chunk signature, and based on this determination, select the bottom-level data structure 226 for placement of manifest 211. In some examples, instructions 122 may use the smallest chunk signature rather than the largest chunk signature. In other examples, instructions 122 may use the identifiers of the bottom-level data structures themselves to determine the affinity, rather than the chunk signatures they contain.For example, instructions 122 may determine from the references 168 of manifest 211 the identifiers of the bottom-level data structures referenced by the references 168 and determine that manifest 211 has an affinity to the referenced bottom-level data structure with the largest identifier, and based on this, select the referenced bottom-level data structure with the largest identifier for placement of manifest 211. In other examples, the referenced bottom-level data structure with the smallest identifier may be selected.
[0069] In some examples, different affinity measures may be used in combination. For example, a binary measure of affinity may be used as a tiebreaker if a manifest has the same degree of affinity according to another measure. For example, if instructions 122 determine, based on a similarity measure (e.g., Jaccard index, etc.), that manifest 211 has the greatest affinity to the two bottom-level data structures 224 and 226 (i.e., the same affinity to both), a maximum or minimum chunk signature or identifier (as described above) may be used to select between the bottom-level data structures 224 and 226 for placement of manifest 211. In other examples, links may be resolved in other ways, such as by selecting a most recently referenced bottom-level data structure (in manifest 211) from among the linking data structures.While the examples described herein may use one or more affinity measures or conditions (alone or in combination), instructions 122 may use a single method (e.g., one measure or combination of measures) of affinity to make placement decisions in deduplication system 100. In this way, the examples described herein may operate to influence placement decisions such that manifests for duplicate data sequences have a greater chance of having the same bottom-level data structure selected for placement, which may provide a greater chance for deduplication of such manifests, as discussed below with respect to . Fig. described. In the examples described herein, each time a new manifest is to be stored (for objects below the threshold size), instructions 122 may select a bottom-level data structure in which to place a manifest, as described above; this may occur upon initial read-in of a new object or upon receipt of additional data for a previously read-in object. For example, when new data is received for an object previously stored in deduplication system 100 via a small object representation, instructions 122 may generate a new manifest to represent the updated version of the object, and in such examples, instructions 122 may select a bottom-level data structure to store the new manifest for the updated version of the object (if the object remains below the threshold size).In such examples, instructions 122 may make the selection described above for storing a manifest when a manifest is created and when a manifest is updated (in examples where updating an object involves creating a new manifest for the updated object).
[0070] In some examples, the small object metadata of the small object representation of the second object 40 may initially be a manifest 211 as a result of the initial ingest of the second object 40. In such examples, the second object 40 is Fig. not an updated (and larger) version of a previously stored object 40, but the second object 40 may be both too large to be stored in a chunk and smaller than the threshold size when initially ingested by the deduplication system 100.
[0071] In such examples, upon initially receiving and processing the second object 40, instructions 122 may determine that the second object 40 is both too large to be stored in a chunk and smaller than the threshold size. Based on the determination that the second object 40 is smaller than the threshold size, instructions 122 may store a small object representation of the second object 40 in the deduplication system 100. In such examples, the small object representation may include the contents of the second object 40 contained in a plurality of chunks (e.g., 201, 203, 204, etc.).The chunks of the second object 40 may include previously stored matching chunks (which are incorporated into the representation of the small object by referencing the previously stored chunks), new chunks (for which no match was found) stored based on the ingest of the object 40, or a combination thereof.
[0072] In such examples, the instructions 122 may select a bottom-level data structure in which to store a manifest 211 for the second object 40 (e.g., based on the affinity described above). In the example of Fig. Instructions 122 may select bottom-level data structure 226 based on appropriate affinity measures between it and manifest 211, and based on the selection, store manifest 211 in bottom-level data structure 226. In such examples, instructions 122 may store in top-level data structure 210 a direct reference 37 from top-level data structure 210 to manifest 211 in bottom-level data structure 226. In such examples, the manifest 211 is the small object metadata of the small object representation of the object 40. The direct reference 37 from the top-level data structure 210 to the manifest 211 omits any intermediate-level data structures that are separate from the top-level and bottom-level data structures of the small object representation (e.g., reference chains or indirect references to or through intermediate-level data structures to access the manifest 211 from the top-level data structure 210).In the top-level data structure 210, the direct reference 37 can be mapped to the ID 30 for the second object 40.
[0073] In such examples, the representation of the small object 40 may include the top-level data structure 210, the direct reference 37 associated with the ID 30 for the second object 40, the bottom-level data structure 226, the manifest 211 of the bottom-level data structure 226, the chunk metadata 150 referenced by the manifest 211, and the chunks referenced by this chunk metadata 150. In the example of Fig. The bottom-level data structure 226 of the small object representation contains chunk metadata 150 for the second object 40, including the respective storage locations for each chunk of the second object 40. As described above, the manifest 211 for the second object 40 represents an ordering of the plurality of chunks for reconstructing the second object 40 via references to the chunk metadata 150 of the respective plurality of chunks of the second object 40.
[0074] In some examples, instructions 122 may change the deduplicated representation of object 40 from a small object representation of the second object 40 (including a direct reference to the metadata of the small object) to a large object representation of the second object 40. In some examples, the deduplicated representation of object 40 may initially be a small object representation that includes a manifest as metadata of the small object (as in the example of Fig. ), and in other examples, the deduplicated representation of the object 40 may initially be a small object representation that includes an instance of chunk metadata 150 as small object metadata.
[0075] For example, after storing the small object representation of the second object 40 with additional content 41 in the deduplication system 100, the instructions 122 may determine that further additional content should be stored in the deduplicated representation of the second object 40 in the deduplication system 100 (see Fig. ). For example, instructions 122 may receive another updated, larger version of second object 40 for storage (i.e., larger than the version of object 40 with additional content 41). Instructions 122 may determine if the received and further updated, larger version of second object 40 is larger than the threshold size. If so, instructions 122 may change the small object representation of second object 40 (including direct reference 37 to manifest 211 of bottom-level data structure 226) to a large object representation of the further updated second object 40. In some examples, the threshold size may be a size at which a given object is too large for deduplication system 100 to represent all chunks of the given object in a single manifest of deduplication system 100.For example, as described above, a manifest for an object (or a portion thereof) may include references to chunk metadata 150 for the chunks 260 of the object (or a portion thereof), and the instructions 122 may determine that the further updated second object 40 is too large for the deduplication system 100 to store all references to chunk metadata 150 for the chunks 260 of the further updated second object 40 in a single manifest.
[0076] In such examples, the determination of whether the object is less than (or greater than or equal to) the threshold size may be made indirectly, as described above, based on the size of the manifest for the object in the deduplicated representation. In other examples, the determination may be made directly by comparing the size of the object (without deduplication) to the threshold size, as described above.
[0077] To change the deduplicated representation of object 40 from a small object representation to a large object representation of the further updated version of second object 40, instructions 122 may add one or more intermediate-level data structures to hierarchy 290. The added intermediate-level data structure(s) may be separate from the top-level data structure 210 and the bottom-level data structure 226 of the small object representation of second object 40, and instructions 122 may add them between the top-level data structure 210 and the bottom-level data structure 226 in hierarchy 290. In the present examples, "between a first and a second data structure in hierarchy 290" means that it is lower than one of the data structures and higher than the other.
[0078] Referring to the example of Fig. Instructions 122 may change the deduplicated representation of object 40 from a small object representation including a manifest 211 in bottom-level data structure 226 to a large object representation of the further updated version of second object 40. In such examples, instructions 122 may update manifest 211 for second object 40 to reflect a reconstruction order for the chunks of the second object with additional content 41 and the further additional content. In such examples, references to instances of chunk metadata 150 for chunks representing the further additional content of object 40 may be added to manifest 211. The chunks representing the further additional content may include previously stored matching chunks, new chunks (for which no match was found) added to deduplication system 100, or a combination thereof.
[0079] The instructions may store the updated manifest 211 in one or more added intermediate-level data structures. For example, instructions 122 may store the manifest 211 in a new intermediate-level data structure (e.g., a similar data structure 216) beneath another intermediate-level data structure (e.g., a similar data structure 212), both of which are located below the top-level data structure 210 in the hierarchy 290. In other examples, instructions 122 may divide the updated manifest 211 into corresponding manifests for separate parts of the further updated second object 40 and store each of the corresponding manifests for the parts in a corresponding one of the added intermediate-level data structures (e.g., such as the corresponding manifests 160 and 164 of the intermediate-level data structures 214 and 216, as described above).Instructions 122 may update direct reference 37 with a reference to the added intermediate-level data structure(s). For example, the updated direct reference may point to an added intermediate-level data structure (e.g., such as data structure 212) located above and pointing to each of the other added intermediate-level data structure(s) (e.g., such as data structures 212, 214, and 216).
[0080] In some examples, the deduplication system 100 may first store a small object representation of the object 40 including the chunk metadata 150 as small object metadata, later convert the small object metadata into a manifest 211, and later change the small object representation including the manifest 211 into a large object representation, as described above. In other examples, the deduplication system 100 may first store a small object representation of the object 40 including the manifest 211 as the small object metadata, and later change the small object representation including the manifest 211 into a large object representation, as described above.
[0081] In other examples, instructions 122 may change the deduplicated representation of object 40 from a small object representation, including an instance of chunk metadata 150 as small object metadata, to a large object representation of an updated version of the second object 40. For example, after storing an initial small object representation of object 40, including an instance of chunk metadata 150 as metadata of the small object, instructions 122 may receive an updated version of object 40 with additional content and store a manifest for the updated second object 40 with the additional content (see Fig. ). In some examples, instructions 122 may determine that the updated second object 40 with the additional content is larger than the threshold size.
[0082] In such examples, the instructions 122 may generate a manifest for the updated second object 40 to represent a reconstruction order for the chunks of the updated second object 40 with the additional content, may add one or more intermediate-level data structures that are separate from and between the top-level and bottom-level data structures of the small object representation of the object 40 (as described above with respect to Fig. described above), and can store the generated manifest in one or more of the added intermediate-level data structures (as described above with respect to Fig. described). In such examples, instructions 122 may update the direct reference of the small object representation with a reference to the added intermediate-level data structure(s), as described above with respect to Fig. described.
[0083] Although examples of transitioning between types of small object metadata and between small and large object representations are described here in the context of the deduplication system 100 receiving updated (larger) versions of an object, the deduplication system 100 may perform similar functions in other contexts. For example, during ingestion of an object, instructions 122 may initially specify that the object be stored with a small object representation and only transition to a large object representation when the amount of data of the ingested object exceeds the threshold size. Instructions 122 may similarly transition from an instance of chunk metadata 150 as small object metadata to a manifest as small object metadata for an object as more data is ingested for a single object during an ingestion process.
[0084] Further examples are given here in relation to Fig. described. Fig. is a block diagram of an exemplary deduplication system 100 for deduplicating manifests. The deduplication system 100 of Fig. is the same as the deduplication system 100 from Fig. , although the example of Fig. some other functionalities of the deduplication system 100 than the examples of Fig. shows.
[0085] In the example of Fig. the deduplication system 100 may store a large object representation of a first object 10 associated with the ID 20 (as described above with respect to Fig. described), and may store a small object representation of a second object 40 associated with the ID 30, including a manifest 211 as small object metadata in the bottom-level data structure 226 (as described above with respect to Fig. described). The manifest 211 includes references 168 to chunk metadata 150 for each chunk of the second object 40. Each instance of chunk metadata 150 referenced by the manifest 211 includes a chunk signature (e.g., SIG-2, SIG-4, SIG-7, etc. in Fig. ). For each instance of chunk metadata 150, the chunk signature of that instance of chunk metadata 150 represents the contents of the chunk 260 associated with that instance of chunk metadata 150.
[0086] In the example of Fig. Instructions 122 may generate a manifest signature 169 ("SIG-11") for the second object 40 based on each of the chunk signatures of the chunk metadata 150 referenced by the manifest 211 for the second object 40 (e.g., based on each of the chunk signatures of the chunk metadata 150 for each chunk 260 of the second object 40). Instructions 122 may store the manifest signature 169 in a bottom-level data structure (e.g., 226) of the small object representation of the second object 40. For example, instructions 122 may store the manifest signature 169 in the manifest 211 of the bottom-level data structure 226.
[0087] In some examples, the instructions 122 may determine the manifest signature 169 by applying a signature function (e.g., a hash function, etc.), as described above, to the chunk signatures for the chunks 260 of the second object 40 contained in the instances of the chunk metadata 150 referenced by the manifest 211. In the example of Fig. the second object 40 in the deduplication system 100 may be represented by a collection of chunks referenced (via chunk metadata) by the manifest 211, including, for example, chunks 201, 203, and 204 (at least).
[0088] Manifest 211 may reference each of these chunks via references to instances of chunk metadata 150 associated with those chunks. For example, manifest 211 includes a reference 168 to chunk metadata 150 with ID 272 associated with chunk 204, a reference 168 to chunk metadata 150 with ID 270 associated with chunk 203, and a reference 168 to chunk metadata 150 with ID 249 associated with chunk 201. These instances of chunk metadata each contain a chunk signature (e.g., SIG-7, SIG-4, and SIG-2), and instructions 122 may determine the manifest signature 169 based on these chunk signatures in the chunk metadata 150 referenced by the manifest 211. For example, instructions 122 may concatenate all chunk signatures of the instances of chunk metadata 150 referenced by the manifest 211 (e.g., chunk signatures SIG-7, SIG-4, SIG-2, etc.).) and then apply the signature function to the result of the concatenation to generate the manifest signature 169. In other examples, the manifest signature can be generated in any other suitable way (e.g., based on the chunk signatures). Although the example in . Fig. with respect to three illustrated references 168 to chunk metadata, in other examples the manifest 211 may contain more or fewer references 168 and correspondingly more or fewer chunk signatures to generate the manifest signature 169.
[0089] Continuing the example of Fig. Instructions 122 may receive a third object 60 for storage (in deduplication system 100) via a deduplicated representation including a plurality of chunks to represent the contents of third object 60. In some examples, instructions 122 may generate a manifest for third object 60, determine that third object 60 is below the threshold size, and, based on that determination, generate a manifest signature for third object 60 based on each of the chunk signatures of the chunk metadata referenced by the manifest of third object 60 (e.g., as described above with respect to manifest signature 169).
[0090] For example, the manifest for the third object 60 includes, for each chunk 260 of the third object 60, a reference to a corresponding instance of chunk metadata 150 containing a chunk signature representing the contents of the associated chunk 260, and the instructions 122 may generate the manifest signature for the third object 60 based on each of these chunk signatures. In some examples, the chunks 260 representing the contents of the third object 60 may already be stored in the deduplication system 100, they may be added during the ingest process for the third object 60, or some combination thereof.
[0091] In some examples, instructions 122 may select a bottom-level data structure in which to store the manifest for the third object 60, as described above. In the example of Fig. Instructions 122 may select bottom-level data structure 226 for storing the manifest. In such examples, before storing the manifest for the third object 60, instructions 122 may compare the manifest signature for the third object 60 with the manifest signature 169 for the second object 40. Based on the determination that the manifest signature 169 for the second object is equivalent to the manifest signature for the third object 60, instructions 122 may (in deduplication system 100) store a small object representation of the third object 60 that includes a direct reference 55 from the top-level data structure 210 to the manifest 211 for the second object 40 in the bottom-level data structure 226.
[0092] In such examples, instructions 122 may perform deduplication at the manifest level. In such examples, before storing a new manifest for the third object 60, instructions 122 may use the respective manifest signatures to determine whether an equivalent manifest is already stored in the deduplication system 100, as in the example of Fig. . If an equivalent manifest is already stored in the deduplication system 100, a reference (e.g., 55) to the already stored manifest may be stored in the deduplication system 100 instead of a duplicate of the manifest. In the example of Fig.A small object ID 50 associated with the third object 60 may be stored in the top-level data structure 210 and associated with the reference 55 to the already stored manifest 211. In such examples, instructions 122 may store a small object representation of the third object 60, including the direct reference 55 from the top-level data structure 210 to the manifest for the second object, in the bottom-level data structure. In the examples described herein, selecting a bottom-level data structure for storing a manifest based on one or more affinity measures, as described above, may result in instructions 122 selecting the same bottom-level data structure for two identical manifests so that they can be deduplicated as described above.In such examples, the use of consistent placement selection criteria is expected to result in identical placement decisions for identical manifest data.
[0093] A "computing device" may be a server, a storage device, a storage array, a backup device, a desktop or laptop computer, a switch, a router, or other processing device or equipment having at least one processing resource. In the examples described herein, a processing resource may include, for example, one or more processors contained within a single computing device or distributed across multiple computing devices. A processor may be at least one of a central processing unit (CPU), a semiconductor-based microprocessor, a graphics processing unit (GPU), a field-programmable gate array (FPGA) configured to fetch and execute instructions, other electronic circuitry capable of fetching and executing instructions stored on a machine-readable storage medium, or a combination thereof.In the examples described herein, a processing resource may retrieve, decode, and execute the instructions stored on a storage medium to perform the functions described with respect to the instructions stored on the storage medium. In other examples, the functions described with respect to the instructions described herein may be implemented in the form of electronic circuits, in the form of executable instructions encoded on a machine-readable storage medium, or a combination thereof. The storage medium may either be located on the computing device that executes the machine-readable instructions, or it may be remote from the computing device but accessible for execution (e.g., via a computer network). In the examples presented herein, a storage medium 120 may be implemented by one or more machine-readable storage media.
[0094] In the examples described herein, a backup appliance, storage array, or the like may be a computing device that includes a plurality of storage devices and one or more controllers that interact with client (or host) devices and control access to the storage devices. In some examples, the storage devices may include hard disk drives (HDDs), solid-state drives (SSDs), or any other suitable type of storage device, or any combination thereof. In some examples, the controller(s) may virtualize the storage capacity provided by the storage devices to allow a host to access a virtual object (e.g., a volume) comprised of storage space from multiple different storage devices.
[0095] A "machine-readable storage medium" may be any electronic, magnetic, optical, or other physical storage medium that contains or stores information such as executable instructions, data, and the like. For example, any machine-readable storage medium described herein may be RAM, EEPROM, volatile memory, non-volatile memory, persistent memory, flash memory, a storage drive (e.g., an HDD, an SSD), any type of storage disk (e.g., a compact disc, a DVD, etc.), or the like, or a combination thereof. Furthermore, any machine-readable storage medium described herein may be non-transmissive. In the examples described herein, a machine-readable storage medium or media may be part of an article (or article of manufacture). An article or article of manufacture may refer to a single manufactured component or to multiple components.In some examples, instructions may be part of an installation package that, when installed, can be executed by a processing resource to implement the functionality described here.
[0096] In the examples described herein, the term "based on" is inclusive and means the same as the alternative phrase "at least based on" or "at least partially based on." In the examples described herein, functionality described as being performed by "instructions" may be understood as functionality that can be performed by those instructions when executed by a processing resource. In other examples, functionality described in terms of instructions may be implemented by one or more machines, which may be any combination of hardware and programming for implementing the functionality of the machine(s).
Claims
[1] A method (300) comprising: Storing (305), in a deduplication system (100), a large object representation of a first object (10) that is larger than a threshold size, the large object representation comprising metadata organized in a hierarchy (290) of data structures comprising: a top-level data structure (210); one or more bottom-level data structures (220, 222, 224, 226), each comprising chunk metadata (150) with storage locations (80, 82, 84, 85, 86, 87) for respective chunks (260) of the first object (10); and one or more intermediate-level data structures (212, 214, 216) that are separate from and between the top-level and bottom-level data structures (220, 222, 224, 226) in the hierarchy (290), comprising a manifest that represents a reconstruction order for the chunks (260) of the first object (10) via references to the chunk metadata (150) of the one or more bottom-level data structures (220, 222, 224, 226); Determining (310) that a second object (40) is smaller than the threshold size; and based on the determination, storing (315) a small object representation of the second object (40) in the deduplication system (100), comprising: a direct reference from the top-level data structure (210) to small object metadata in a bottom-level data structure of the small object representation, wherein the direct reference omits all intermediate-level data structures that are separate from and lie between the top-level data structure and the bottom-level data structure of the small object representation, wherein the bottom-level data structure of the small object representation comprises chunk metadata for the second object (40) including a respective storage location (80, 82, 84, 85, 86, 87) for each chunk of the second object (40). [2] The method (300) of claim 1, wherein: a content of the second object (40) is contained in a single chunk (204); the small object metadata comprises the chunk metadata (150) for the individual chunk (204) of the second object (40); and the direct reference comprises a reference to the chunk metadata (150) for the individual chunk (204) of the second object (40), the reference being stored in the top-level data structure (210). [3] The method (300) of claim 1, wherein: a content of the second object (40) is contained in a plurality of chunks (201, 203, 204); the small object metadata comprises a manifest for the second object (40) stored in the bottom-level data structure of the small object representation; and the manifest for the second object (40) represents an order of the plurality of chunks (201, 203, 204) for reconstructing the second object (40) via references to the chunk metadata of the respective plurality of chunks (201, 203, 204) of the second object (40). [4] The method (300) of claim 3, wherein: a deduplication domain of the deduplication system (100) comprises a plurality of bottom-level data structures, including the one or more bottom-level data structures (220, 222, 224, 226) of the large object representation; and Storing the small object representation of the second object (40) in the deduplication system (100) comprises: Selecting a given bottom-level data structure from the plurality of bottom-level data structures as a bottom-level data structure of the small object representation and storing the manifest based on an affinity between the manifest and the given bottom-level data structure. [5] The method (300) of claim 4, wherein for each chunk of the second object (40), the respective chunk metadata comprises a chunk signature representing a content of the chunk, the method further comprising: Generating a manifest signature for the second object (40) based on each of the chunk signatures of the chunk metadata referenced by the manifest of the second object (40); and Store the manifest signature in the bottom-level data structure of the small object representation. [6] The method (300) of claim 5 further comprising: Creating a manifest for a third object (60) comprising, for each chunk of the third object, a reference to respective chunk metadata including a chunk signature representing the contents of the chunk; and based on a determination that a third object (60) is below the threshold size and includes content to be represented by a second plurality of chunks: Generating a manifest signature for the third object (60) based on each of the chunk signatures of the chunk metadata referenced by the manifest of the third object (60). [7] The method (300) of claim 6 further comprising: Comparing the manifest signature for the third object (60) with the manifest signature for the second object (40); and based on a determination that the manifest signatures for the second and third objects (60) are equivalent, storing a small object representation of the third object (60) in the deduplication system (100) that includes a direct reference from the top-level data structure (210) to the manifest for the second object (40) in the bottom-level data structure. [8] An article comprising a non-transitory, machine-readable storage medium (120) comprising instructions executable by at least one processing resource (110) to: based on a determination that a first object (10) is larger than a threshold size, store a large object representation of the first object (10) in a deduplication system (100), the large object representation comprising metadata organized in a hierarchy (290) of data structures comprising: a top-level data structure (210); one or more bottom-level data structures (220, 222, 224, 226) each comprising chunk metadata (150) with storage locations (80, 82, 84, 85, 86, 87) for respective chunks (260) of the first object (10); one or more intermediate-level data structures (212, 214, 216) that are separate from and between the top-level and bottom-level data structures (220, 222, 224, 226) in the hierarchy (290), comprising a manifest that represents a reconstruction order for the chunks (260) of the first object (10) via references to chunk metadata (150) of the one or more bottom-level data structures (220, 222, 224, 226); and wherein the one or more intermediate-level data structures (212, 214, 216) are separated from the top-level and bottom-level data structures (220, 222, 224, 226) and lie between them in a data structure hierarchy; and based on a determination that a second object (40) is smaller than the threshold size, storing in the deduplication system (100) a small object representation of the second object (40) comprising: a direct reference from the top-level data structure (210) to small object metadata in a bottom-level data structure of the small object representation, wherein the direct reference (35) omits all intermediate-level data structures that are separate from and lie between the top-level data structure and the bottom-level data structure of the small object representation, wherein the bottom-level data structure of the small object representation comprises chunk metadata for each chunk of the second object (40) including a respective storage location (80, 82, 84, 85, 86, 87) for each chunk of the second object (40). [9] The article of claim 8, wherein the instructions are executable to: based on a determination that a content of the second object (40) is small enough to be represented in the deduplication system (100) by a single chunk (204): Storing a reference to the chunk metadata for the individual chunk (204) of the second object (40) in the top-level data structure (210), wherein the chunk metadata for the individual chunk (204) is stored in the bottom-level data structure of the small object representation; wherein the direct reference (35) comprises the reference to the chunk metadata (150) for the individual chunk (204). [10] The article of claim 9, wherein the instructions are executable to: after storing the small object representation of the second object (40) in the deduplication system (100), determine that additional content to the deduplicated representation of the second object (40) should be stored in the deduplication system (100); based on a determination that the content of the second object (40) with the additional content is too large to be stored in a chunk and is smaller than the threshold size: storing a manifest for the second object (40) with the additional content in the bottom-level data structure of the small object representation, wherein the manifest for the second object (40) represents, via references to chunk metadata, a reconstruction order of a first plurality of chunks containing the content of the second object (40); and modify the top-level data structure of the small object representation to include a reference to the manifest for the second object (40) in the bottom-level data structure of the small object representation as a direct reference. [11] The article of claim 10, wherein the instructions are executable to: after storing the small object representation of the second object (40) with the additional content in the deduplication system (100), determine that further additional content to the deduplicated representation of the second object (40) should be stored in the deduplication system (100); based on a determination that the content of the second object (40) with the further additional content is larger than the threshold size, change the small object representation of the second object (40) to a large object representation of the second object; The instructions for changing include instructions for: Adding one or more intermediate-level data structures that are separated from the top-level and bottom-level data structures of the small object representation and lie between them in a data structure hierarchy; Updating the manifest for the second object (40) to represent a reconstruction order for the chunks of the second object (40) with the additional content and the further additional content; and Storing the updated manifest in the one or more intermediate-level data structures; and Update the direct reference with a reference to the added one or more intermediate-level data structures. [12] The article of claim 8, wherein the instructions are executable to: based on a determination that the content of the second object (40) is not small enough to be represented in the deduplication system (100) by a single chunk (204): storing a manifest for the second object (40) in the bottom-level data structure of the small object representation, wherein the manifest for the second object (40) represents, via references to chunk metadata, a reconstruction order of a first plurality of chunks containing the content of the second object (40); and to store a reference to the manifest for the second object (40) in the bottom-level data structure of the small object representation in the top-level data structure (210) as a direct reference. [13] The article of claim 12, wherein the instructions are executable to: after storing the small object representation of the second object (40) in the deduplication system (100), determine that additional content to the deduplicated representation of the second object (40) should be stored in the deduplication system (100); based on a determination that a content of the second object with the additional content is larger than the threshold size, change the small object representation of the second object (40) to a large object representation of the second object (40); The instructions for changing include instructions for: Adding one or more intermediate-level data structures that are separate from the top-level and bottom-level data structures of the small object representation and lie between them in a data structure hierarchy; Updating the manifest for the second object (40) to represent a reconstruction order for the chunks of the second object (40) with the additional content; Storing the updated manifest in the one or more intermediate-level data structures; and Update the direct reference with a reference to the added one or more intermediate-level data structures. [14] A computer device comprising: at least one processing resource (110); and a non-transitory, machine-readable storage medium (120) comprising instructions executable by the at least one processing resource (110) to: storing a large object representation of a first object (10) that is larger than a threshold size in a deduplication system (100), the large object representation comprising metadata organized in a hierarchy (290) of data structures comprising: a top-level data structure (210); one or more bottom-level data structures (220, 222, 224, 226), each comprising chunk metadata (150) with storage locations (80, 82, 84, 85, 86, 87) for respective chunks (260) of the first object (10); and one or more intermediate-level data structures (212, 214, 216) that are separate from and between the top-level and bottom-level data structures in the hierarchy (290), comprising a manifest that represents a reconstruction order for the chunks (260) of the first object (10) via references to chunk metadata (150) of the one or more bottom-level data structures (220, 222, 224, 226); to determine that a second object (40) is smaller than the threshold size; and based on a determination to store a small object representation of the second object (40) in the deduplication system (100), comprising: a direct reference (35) from the top-level data structure (210) to small object metadata in a bottom-level data structure of the small object representation, wherein the direct reference (35) omits all intermediate-level data structures that are separate from and lie between the top-level data structure and the bottom-level data structure of the small object representation, wherein the bottom-level data structure of the small object representation comprises chunk metadata for the second object (40) comprising a respective storage location (80, 82, 84, 85, 86, 87) for each chunk of the second object (40). [15] The computing device of claim 14, wherein: a content of the second object (40) is contained (204) in a single chunk; the small object metadata comprises the chunk metadata (150) for the individual chunk (204) of the second object (40); the direct reference (35) comprises a reference to the chunk metadata (150) for the individual chunk (204) of the second object (40) in the bottom-level data structure of the small object representation; and the direct reference is stored in the top-level data structure of the small object representation. [16] The computing device of claim 14, wherein: a content of the second object (40) is contained in a plurality of chunks (201, 203, 204); the small object metadata comprises a manifest for the second object (40) stored in the bottom-level data structure of the small object representation; and the manifest for the second object (40) represents a reconstruction order for the plurality of chunks (201, 203, 204) for the second object (40) via references to the chunk metadata of the plurality of chunks (201, 203, 204) of the second object (40). [17] The computing device of claim 16, wherein the manifest for the second object (40) comprises references to chunk metadata of a plurality of different bottom-level data structures of the deduplication system (100). [18] The computing device of claim 17, wherein for each chunk of the second object, the respective chunk metadata comprises a chunk signature representing a content of the chunk, the instructions further comprising instructions executable by the at least one processing resource (110) to: Generating a manifest signature for the second object (40) based on each of the chunk signatures of the chunk metadata referenced by the manifest for the second object (40); and Store the manifest signature in the bottom-level data structure of the small object representation. [19] The computing device of claim 18, wherein the instructions further comprise instructions executable by the at least one processing resource (110) to: Creating a manifest for a third object (60) comprising, for each chunk of the third object (60), a reference to respective chunk metadata including a chunk signature representing the contents of the chunk; and based on a determination that a third object (60) is below the threshold size and includes content to be represented by a second plurality of chunks: Generating a manifest signature for the third object (60) based on each of the chunk signatures of the chunk metadata referenced by the manifest of the third object (60). [20] The computing device of claim 19, wherein the instructions further comprise instructions executable by the at least one processing resource (110) to: Comparing the manifest signature for the third object (60) with the manifest signature for the second object (40); and based on a determination that the manifest signatures for the second and third objects (60) are equivalent, storing a small object representation of the third object in the deduplication system (100) that includes a direct reference from the top-level data structure to the manifest for the second object (40) in the bottom-level data structure.
Citation Information
Patent Citations
Method for handling large object files in an object storage system
US20120233522A1
Data deduping in content centric networking manifests
US20170270134A1
Chunk storage deduplication using graph structures
US20180218005A1
Backup item metadata including range information
US20190121705A1
