Shallow cache for content replication

Abstract

Embodiments relate to efficiently copying data from a source storage space to a target storage space. The storage spaces share a common namespace of paths at which content units are stored. A shallow cache is maintained for the target storage space. Each entry in the cache includes a hash of a content unit in the target storage space and an associated hierarchical path in the target storage space at which the corresponding content unit is stored. When a set of content units in the source storage space is copied to the target storage space, any content unit with a hash in the cache is copied from one of the associated paths in the cache, thereby avoiding having to copy content from the source storage space.

Description

Background Art

[0001] A computing system stores and manages a hierarchy of content units or data items. Each content unit in this hierarchy or storage space has a path or location. The paths are all in the same namespace, and the content units can be accessed by referencing their paths. Often, portions of one storage space need to be copied to another storage space. That is, a collection of content units at corresponding paths in a source storage space may need to be copied to a target storage space that shares the same namespace. For example, a directory in a remote file system may need to be copied to a directory in a local file system.

[0002] Depending on the type of application or storage device, the same instance of a content unit may be stored in multiple paths in the storage space. In the case of a file system file, there may be multiple files containing the same content but at different full paths in the file system. For example, the same file content "X" may be stored at "\A\B" and at "\A\C\D". Files "B" and "D" store the same content, but at different locations and with different file names.

[0003] When a set of paths and corresponding content units are copied from a source storage space to a target storage space, the copying of content units may have inefficiencies. The same content may be transferred from the source to the target multiple times. Referring to the example above, when files "B" and "D" contain the same content, it would be redundant and wasteful to copy both files "B" and "D" from the source to the target. As only the inventors have observed, another inefficiency may occur when the content units of the portion of the source storage space to be transferred to the target storage space already exist at the target storage space. For example, if the content units at "\A\B" and "\A\C" may be transferred as a package, if the target storage space already contains the same content as, for example, "\A\B", but at a different local location (such as "\D\E\F"), the inventors have observed that since the same content is already available in the target storage space (at "\D\E\F"), the content units transferred at "\A\B" are potentially avoidable.

[0004] The techniques discussed below relate to using a shallow cache to efficiently transfer packages or collections of content units or data items from a source storage space to a target storage space by leveraging existing content at the target storage space. Summary of the invention

[0005] The following Summary is included to introduce only some of the concepts discussed in the following Detailed Description. This Summary is not extensive and is not intended to delineate the scope of the claimed subject matter, which is set forth by the appended claims.

[0006] Embodiments relate to efficiently copying data from a source storage space to a target storage space. The storage spaces share a common namespace for storing paths of content units. A shallow cache is maintained for the target storage space. Each entry in the cache includes a hash of a content unit in the target storage space and an associated hierarchical path where the corresponding content unit is stored in the target storage space. When a set of content units in the source storage space is to be copied at the target storage space, any content unit with a hash in the cache is copied from one of the associated paths in the cache, thereby avoiding having to copy the content from the source storage space.

[0007] Many of the accompanying features will be explained below with reference to the following detailed description considered in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present specification will be better understood from the following detailed description read in light of the accompanying drawings, wherein like reference numerals are used to refer to like parts throughout the appended description.

[0009] Figure 1 A system for copying a data set of data items from a source storage space to a target storage space is shown.

[0010] Figure 2 Additional details of the dataset are shown.

[0011] Figure 3 The process for constructing the dataset is shown.

[0012] Figure 4 The target storage space is shown supplemented by a shallow cache.

[0013] Figure 5 A process for initializing a shallow cache is shown.

[0014] Figure 6 A process for copying a data set to a target storage space using a shallow cache is shown.

[0015] Figure 7 An example of initializing a copy of a shallow cache is shown.

[0016] Figure 8 An example of using a shallow cache for replicating a data set is shown.

[0017] Fig. 9 Details of a computing device are shown upon which embodiments described herein may be implemented. DETAILED DESCRIPTION

[0018] Figure 1 A system for copying a data set 100 of data items from a source storage space 100 to a target storage space 102 is shown. Data items or content units can be discrete units of any type of data, such as files, memory blocks, media data, etc., as long as they are individually distinguishable and accessible. They can have variable lengths or uniform sizes. Storage spaces 102 / 104 can be co-located on the same computing device or they can be on corresponding different computing devices connected by a network. It is worth noting that they share the same namespace. In addition, storage spaces typically organize their content units in a hierarchical manner. Content units in a storage space can be arranged or located by names, paths, identifiers of tree nodes, etc. The term "path" will be used below to refer to these types of hierarchical location descriptors.

[0019] A common operation is to copy portions of a source storage space 102 to a target storage space 104. Conceptually, this involves copying a data set 100 that includes a collection of paths in the source storage space 102 and corresponding content units residing at the paths in the source storage space 102. In the case of a file system, the data set 100 may be a collection of entire directories or discrete files.

[0020] In a typical system, the data set 100 or other structure of data for transmission may involve a software component such as a data set builder 106 on the source side, which collects the necessary data and information to allow the receiver to reconstruct the portion of the source data being prepared. Any known packaging tool can be used. As described in detail later, the data set 100 can be a combination of indicia of portions of the source data, such as a list of content unit paths. The data set 100 can optionally include copies of the content units. In one embodiment, a list is used, and the target side can use the list to decide which content units to pull as needed.

[0021] The data set builder 106 can perform a build process 108, which includes steps such as receiving a request to build a data set and then building the data set. The request can specify a set of paths, a root path or a subtree, etc. In one embodiment, as discussed in more detail below, building the data set 100 includes identifying content items with the same content, providing an identifier (e.g., a hash) for the content, and adding the identifier or hash and the corresponding source path containing the same corresponding content. In the case of a file system, the data set 100 is a list of each unique hash in the data set, and each hash is accompanied by a list of the full path names of the files with the same hash.

[0022] In a pull scenario where the target side pulls data from the source side, the data replicator process 100 or the like performs the replication process 112. The data replicator process 100 may include steps such as selecting a data set 100, providing a request for the data set 100 to the source side, replicating at least a tag of the content in the data set, and then replicating the relevant content units from the source storage space 102 and distributing them to the correct namespace location or path in the target storage space 104.

[0023] Figure 1 The system shown is only an example. In different embodiments, the steps can be performed in different orders or at different locations. All construction and replication can occur on the source side or on the target side. Construction and replication can be performed in a combined form.

[0024] Figure 2 Additional details of the data set are shown. As noted, the source storage space 102 stores content units 120. In one embodiment, the content units 120 may be stored in a hierarchy (not shown) that uses the same (or overlapping) namespace as the hierarchy of the target storage space 104. The data set builder 106 may start with a list of paths of content units to be built or packaged. A hash function 122 is applied to each content unit 120 at each path in the list to create a hash 121 for the corresponding content unit. The hash may be performed in advance on all content units in the source storage space. In either case, the data set builder 106 builds a list 124 by merging the hashes of the listed content units. Specifically, any content unit with the same hash contains the same content. Such a hash is added to the list 124 together with a collection of all paths of the corresponding content units with the same hash in the source storage space. The result is that the list 124 represents content units with the same content (and hash value) in groups.

[0025] In one embodiment, data set 100 may include copies of related content units 120 in data sections 125. To minimize size, only one copy of a content unit having the same content is included.

[0026] The target storage space 104 stores content units in an organized hierarchy of paths 126. Figure 2In the example shown, path "A / B" is the complete path or location in the namespace tree of the content unit at node B. In the case of a file system, path 126 is the complete or absolute path in the file system. Using the example path "A / B", a file named "B" is stored at that path. Path 126 can also be a relative path that shares a common sub-root. Once the dataset replicator 110 is instructed which dataset 100 to replicate, the dataset replicator 110 accesses the list 124. As described below, the dataset replicator 110 then determines which content units need to be transferred from the source storage space 102 or from the data segment 125, depending on the actual situation. Regardless of the source, including the copy from the target storage space 104, the list of paths and corresponding content units are copied in the target storage space 104. Each path included in the dataset 100 and its corresponding content unit are copied in the namespace of the target storage space 104.

[0027] Figure 3 A process for creating a data set is shown. At step 140, a request to form a data set is received. The request includes a list of paths of content units or a reference to a content unit. At step 142, a hash is calculated or obtained for each content unit in the request list. Any known hash function can be used to calculate the hash. At step 144, for each unique hash in the data set, a list of corresponding paths in the source data is compiled. At step 146, a manifest 124 is constructed with each content unit in the request list represented as at least one path associated with the corresponding hash.

[0028] Figure 4The target storage space 104 is shown to be supplemented by a shallow cache 160. The shallow cache 160 need not cache content units. Instead, the shallow cache 160 tracks paths in the target storage space 104 that contain the same content unit. This can be in the form of a hash table or hash file, where each hash has an associated set of paths. The set of paths of the hashes in the shallow cache indicates every location in the target storage space 104 where the same content may be located. In some embodiments, the shallow cache does not need to be strictly maintained. It may not be necessary for the shallow cache 160 to continuously reflect the actual contents of the target storage space 104. For example, if a content unit listed in the shallow cache 160 is deleted, the shallow cache 160 does not need to be updated. If a content unit is added, the shallow cache 160 also does not need to be updated. Alternatively, the shallow cache 160 may be kept updated by potential implementations of the target storage space 104 (ie, storage update operations include cache updates) or by learning the accuracy of the shallow cache through its use.

[0029] In fact, the shallow cache 160 indicates that at least some of the content units are likely to be available in the target storage space 104. As discussed below, the shallow cache 160 can be used to improve the efficiency of copying a data set from a source storage space to a target storage space by allowing the content units being copied to be obtained from the target storage unit rather than from the source side. As discussed below, when copying a data set to the target storage space, the shallow cache can be updated; new hashes for inbound content units can be added if they do not already exist, and if they already exist, the corresponding set of paths can be expanded to include any new locations to which such content units are added.

[0030] Figure 5A process for initializing a shallow cache 160 is shown. The shallow cache 160 can be initialized in a variety of ways. In one embodiment (not shown), a complete scan of all content units can be performed in the target storage space. The hash of each content unit can be calculated, and the same set of paths for each unique hash is constructed. For large content, this may be prohibited. Another solution is to use the contents of the manifest to build a shallow cache 160 as the corresponding data set is added to the target storage space. At step 170, the data set being copied is copied from the source side in its entirety. Since the shallow cache is empty initially, when this first data set is copied, it is not known whether any local copies of the content units will be copied. Therefore, all content units of the content units of the data set are copied from the source. At step 172, the content units from the source are added to the local storage space as indicated by the manifest of the data set. At step 174, the mapping of the hashes in the manifest to the set of paths is stored in the shallow cache 160. In one embodiment, the manifest itself is only stored as an initial shallow cache. If the storage space is implemented as a file system file, the shallow cache may be stored in a special directory and initialized by storing a manifest in the special directory.

[0031] Figure 6 A process for using a shallow cache to copy a data set to a target storage space is shown. It is assumed that the shallow cache has been initialized and contains a hash and a corresponding path list of content units in the target storage space. At step 190, a request to copy a data set is received. At step 192, a list of data sets is copied or downloaded.

[0032] At step 194, the data set is copied to the target storage space using the following algorithm. The hashes in the list are traversed. For each hash value H in the list, the existence of H in the shallow cache memory is checked. If H does not exist, the source for the content of H is set to the source storage space. However, if H is found in the shallow cache memory, each path in the path list associated with H in the shallow cache memory is checked to see if it exists, and whether there is a valid copy of the corresponding content unit at the path. The first valid copy found is then set as the source, which will be used to obtain the corresponding content unit. Ultimately, no matter which source will be used for the content unit represented by the current H (i.e., the source storage space or the path in the target storage space), the content unit is copied from the source to each path associated with H in the list. In short, for each unique unit of content to be copied, if possible, the content unit is obtained from the target storage space according to the shallow cache memory, and otherwise the content unit is obtained from the source side. And, no matter where the content unit is retrieved from, once it is retrieved, it is copied to the path to that content unit (hash) indicated by the manifest.

[0033] The shallow cache may also be updated during step 194. Any paths found to not exist in the shallow cache may be deleted from the shallow cache. If a content unit at a path in the shallow cache is found to have a hash that does not match the hash in the shallow cache, then that path in the target storage space is added to the cache (at an existing or new hash / entry, as appropriate).

[0034] At step 196, the list of the copied data set is used to update the shallow cache memory. The details of this step will depend on how the shallow cache memory is implemented. In the file system implementation, the list can be added to the shallow cache memory directory. In another embodiment, the content of the list is logically merged with the shallow cache memory. This can involve adding new paths to the existing cache memory entries (hash path list pairs), and at these new paths, the content units have been added. In another embodiment, the list or units of cache memory data can be periodically merged.

[0035] Figure 7An example of initializing the copying of a shallow cache memory is shown. Manifest 124 includes two manifest entries for two corresponding content units. Content unit 1 200 is represented by a manifest entry including a hash value Hash1, which is associated with paths p1 and p2 ("\A\B" and "\A\B\C\D", respectively). Content unit 2 202 is represented by a manifest entry including a hash value Hash2, which is associated with path p3 ("\A\B\K\L\M\N\O"). According to the paths in the manifest, content unit 1 200 is copied from the source side to paths p1 and p2 in the target storage space, and content unit 2 202 is copied from the source side to path p3 in the target storage space. The manifest or the hash-path data therein is the initial shallow cache memory data.

[0036] Figure 8 shows an example of using a shallow cache for replicating a data set. This example builds on Figure 7 . In this example, manifest 124 includes entries representing content unit 3 and content unit 4. As shown in the manifest, content unit 3 is represented by hash 3 associated with path 4 and path 5, and content unit 4 is represented by hash 1 associated with path 6. First, the manifest entry for content unit 3 204 is processed. The presence of hash 3 is checked in shallow cache 160. Because it is not found, content unit 3 is copied from the source side to path p4 and path p5, and hash 3 is added to the shallow cache along with paths p4 and p5 (which now reflects the added content).

[0037] Then, the manifest entry for content unit 4 is processed. Hash 1 is searched and found in the shallow cache memory. The path in the corresponding cache memory entry is selected (e.g., randomly), in this example, path p2 is selected from the cache memory entry of Hash 1. The content unit stored at path p2 in the target storage space is copied to the new path p6 ("\Q\R\S\T\A\B\C") as specified in the manifest (if other paths are specified, they will also be created and receive copies). Therefore, copies from the source side are avoided. In addition, path p6 is added to the cache memory entry of Hash 1. If path p2 does not exist or does not contain a content unit that is hashed to Hash 1, then p2 may be deleted from the Hash 1 cache memory entry, and another path (e.g., p1) in the Hash 1 cache memory entry may be tried. If there is no valid path, the cache memory entry may be deleted and content unit 4 may be copied from the source side.

[0038] As noted above, in one embodiment, new cache entries are added without regard to possible hash duplication, and deduplication is performed when convenient.

[0039] As also noted above, storage spaces can be implemented in many forms. They can be file systems, data structures in the same virtual or physical memory space, cloud storage services, database elements, etc. Replication can be, for example, between devices via a network, or between storage spaces on the same device.

[0040] Fig. 9 Details of a computing device 300 on which the embodiments described above may be implemented are shown. The technical disclosure herein is sufficient for a programmer to write software, and / or configure reconfigurable processing hardware (e.g., field programmable gate array (FPGA)), and / or design an application specific integrated circuit (ASIC), etc. to run on the computing device 300 (possibly via a cloud API) to implement the embodiments described herein.

[0041] The computing device 300 may have one or more displays 322, a network interface 324 (or several), and storage hardware 326 and processing hardware 328, which may be any one or more of the following combinations: a central processing unit, a graphics processing unit, an analog-to-digital converter, a bus chip, an FPGA, an ASIC, an application-specific standard product (ASSP), a complex programmable logic device (CPLD), etc. The storage hardware 326 may be any combination of the following: a magnetic storage device, a static memory, a volatile memory, a non-volatile memory, an optical or magnetically readable substance, etc. The term "storage device" as used herein does not refer to the signal or energy itself, but to the state of a physical device or substance. The hardware elements of the computing device 300 may cooperate in a manner well understood in the field of machine computing. In addition, an input device may be integrated with the computing device 300 or communicate with the computing device 300. The computing device 300 may have any form factor or may be used in any type of enclosure. The computing device 300 may be in the form of a handheld device such as a smart phone, a tablet computer, a gaming device, a server, a rack-mounted or backplane-mounted computer, a system-on-chip, or other.

[0042] The embodiments and features discussed above may be implemented in the form of information stored in a volatile or non-volatile computer or device readable medium. This is considered to include at least the following media: such as optical storage devices (e.g., compact disk read-only memory (CD-ROM)), magnetic media, flash read-only memory (ROM), or any current or future device for storing digital information. The stored information can be in the following form: machine executable instructions (e.g., compiled executable binary code), source code, byte code, or any other information that can be used to support a computing device or configure a computing device to perform the various embodiments discussed above. This is also considered to include at least volatile memory (such as random access memory (RAM)) and / or virtual memory that stores information (such as central processing unit (CPU) instructions) during the execution of the program that implements the embodiment, and non-volatile media that stores information that allows programs or executable files to be loaded and executed. The embodiments and features can be executed on any type of computing device, including portable devices, workstations, servers, mobile wireless devices, etc.

Claims

1. A method for content replication performed by a computing device, the computing device comprising processing hardware and storage hardware, the method comprising: storing in the storage hardware a local data repository including local data items, the local data items having corresponding unique storage paths, the local data items being stored in the local data repository at the storage paths; maintaining a shallow cache memory comprising cache memory entries, each cache memory entry representing a respective local data item in the local data repository, each cache memory entry comprising a hash of the correspondingly represented local data item and comprising a set of paths at which respective copies of the local data items are stored in the local data repository; determining to add a data set from a remote device to the data repository, the data set represented by a manifest comprising a list of manifest entries, each manifest entry comprising a hash of a corresponding remote data item at the remote device and a set of paths at which corresponding copies of the remote data items are stored on the remote device; Based on determining to add the data set from the remote device to the data repository, adding the data set by: for each manifest entry in the manifest, determining whether any cache entry contains the hash of the manifest entry; as well as When a cache entry is determined to contain a hash of a manifest entry, copying a local data item from a path in the set of paths of the cache entry to each of the paths in the set of paths of the manifest entry; wherein when it is determined that no cache entry in the shallow cache memory contains the hash of the manifest entry, copying the corresponding remote data item to the local data repository at the path in the set of paths of the manifest entry; and updating the shallow cache using the list; Data items are allowed to be deleted from the local data store without a corresponding update to the shallow cache memory.

2. The method according to claim 1 also includes determining that a given hash in the list exists in a given cache memory entry in the shallow cache memory, determining that each of the paths in the set of paths in the given cache memory entry does not contain a given data item corresponding to the given hash, and based on this, copying the given data item from the remote device to the local data repository and updating the shallow cache memory accordingly.

3. The method of claim 1, wherein the local data repository comprises a file system, the data items in the data repository comprise files in the file system, and the paths in the set of paths in the cache memory entries comprise file paths in the file system.

4. The method of claim 1, further comprising determining to copy another data set from the remote device, accessing a second manifest corresponding to the another data set, and adding a hash and path from the second manifest to the shallow cache memory.

5. The method of claim 1, further comprising receiving a plurality of manifests for respective data sets, and merging paths of the manifests associated with the same hash in the manifests.

6. A computing device for content replication, comprising: processing hardware configured to interoperate with the storage hardware; The storage hardware stores instructions, the instructions being configured to, when executed by the processing hardware, cause the computing device to perform a process, the process comprising: providing a content-addressable storage device storing a hierarchy of files, each file being stored at a respective unique path in the hierarchy of files, wherein instances of the same file are stored at different respective unique paths in the hierarchy of files; maintaining a shallow cache memory comprising hashes of files in the hierarchy of files, the shallow cache memory further comprising path lists respectively associated with the hashes, each path list associated with a respective hash comprising a list of one or more paths in the hierarchy of files storing one or more respective instances of the files corresponding to the hashes, wherein at least some of the path lists comprise a plurality of paths; and adding files to corresponding new paths in the content addressable storage by: determining whether a hash of the file exists in the shallow cache memory each time a file is to be added to one or more new paths in the content addressable storage, wherein when the hash of the file is determined to exist in the shallow cache memory, the file is copied from the paths in the corresponding path list to each of the corresponding new paths; wherein when the hash of a file is determined not to exist in the shallow cache, the file is copied from a remote source to each of the new paths corresponding to the file, the remote source providing the new paths; and updating the shallow cache using a list; Files in the content addressable storage device that have corresponding hash and path lists in the shallow cache memory are allowed to be deleted without correspondingly updating the shallow cache memory.

7. A computing device according to claim 6, wherein when the path list in the shallow cache memory is used to add a new instance of the corresponding file to the content addressable storage device, a path in the path list is determined to not exist and based on this, another path in the path list is used as a source to add the new instance of the corresponding file.

8. The computing device of claim 6, wherein the file added to the content addressable storage device is obtained from a remote server via a network, and wherein the remote server is part of a cloud service and the added file is identified by the cloud service.

9. The computing device of claim 6, wherein the process further comprises, when adding a new file to the content addressable storage device using the shallow cache memory, determining that a path in a path list in the shallow cache memory is invalid and removing the path from the path list based on this.

10. A computing device according to claim 7, wherein when a file and a corresponding new path are added to the content-addressable storage device, each of the new paths is added to the content-addressable storage device and each of the new paths added includes a location where an instance of the file is stored.

11. A computer-readable storage hardware storing instructions, the instructions being configured to cause a computing device to perform a process, the storage hardware not including signals, the process comprising: maintaining a storage device including content units, each content unit being stored at a full path, each full path belonging to the same namespace, wherein some of the full paths contain instances of the same corresponding content unit; maintaining a location cache indicating which content units are stored at which full paths in the storage device, wherein given an arbitrary content unit, the full path at which a corresponding instance of the arbitrary content unit is stored in the storage device can be retrieved based on a hash of the arbitrary content unit; receiving a first request to add a first target content unit at a first target full path in the namespace of the storage device; responding to the first request by: obtaining a first target hash of the first target content unit, obtaining a source complete path from the location cache based on the target hash, and using the obtained source complete path to copy the content unit at the source complete path to each of the target complete paths; receiving a second request to add a second target content unit to the namespace of the storage device at a second target full path, determining that the location cache does not contain a valid location for the second target content unit, and based thereon copying the second target content unit from a remote storage device to the storage device at the second target full path; and updating the location cache; Wherein the location cache includes hashes of corresponding content units, and wherein the source full path is retrieved from the location cache by searching the location cache for the first target hash.

12. The computer-readable storage hardware of claim 11, the process further comprising receiving a second request to add a second target content unit at a second target full path in the namespace of the storage device, determining that the location cache memory contains a candidate full path corresponding to a hash of the second target content unit, and calculating a hash of the content unit stored at the candidate full path.

13. The computer-readable storage hardware of claim 12, the process further comprising determining that the calculated hash of the content unit at the candidate full path does not match the hash of the second target content unit, and based on this determining not to use the content unit at the candidate full path to satisfy the second request.

14. The computer-readable storage hardware of claim 11, the set of complete paths in an entry in the cache memory comprising paths that do not contain content units that match the hash associated with the entry.

Images