Garbage Collection for Deduplication Cloud Tiering

Abstract

Systems and methods for performing data protection operations are provided, the data protection operations including garbage collection operations and copy-forward operations. For deduplicated data stored in cloud-based storage or cloud tiering (storing containers containing dead segments and live segments or dead and live regions (such as compressed regions)), dead compressed regions are deleted by copying live compressed regions into new containers and then deleting the old containers. The copy-forward is based on a scheme from the data protection system and is performed using a serverless approach.

Description

Technical Field

[0001] Embodiments of the present invention relate to systems, methods, and devices for protecting data. More particularly, embodiments of the present invention relate to data protection operations, including backup operations, restore operations, deduplication operations, and garbage collection operations. More specifically, embodiments relate to systems and methods for collecting garbage in deduplication cloud tiering. Background Art

[0002] In many computing systems, data is often stored locally or on-site in an on-premises network. For many reasons, some entities desire to move at least some of their data to a target site, such as a cloud-based storage system (the cloud). In order to reduce the storage costs associated with the cloud, it is useful to deduplicate the data before it is written to the cloud. An entity that stores its data in the cloud in a deduplicated manner may expect that its cloud storage costs will decrease as data is deleted. This is not always the case.

[0003] It turns out that deleting objects can have a cost. This is because of data deduplication. When a new object is written to the cloud, part of the data in the previous object may be referenced by the new file. When the old object is deleted, not everything in the object becomes unreferenced. In order to delete the unreferenced parts, the objects need to be defragmented by reading the referenced parts of the objects and writing them to the new objects. While writing to the cloud is cheap and often free, reading data from the cloud can be expensive. More specifically, in the context of a regular garbage collection operation, large amounts of data need to be transferred. Reading and transferring data is expensive.

[0004] For example, the cost of transferring data from a cloud service provider may be $0.09 per GB for the first 10 TB transferred. The cost of reading 1 TB per week is approximately $4,800 per year. An entity deleting many TBs of data may incur high network costs during garbage collection operations, at least because of the need to transfer the data back to the local system in order to properly delete the data stored in a deduplicated form. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] In order to describe the manner in which at least some of the advantages and features of the present invention can be obtained, a more particular description of embodiments of the present invention will be presented with reference to specific embodiments of the invention shown in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the invention and are not therefore to be considered limiting of its scope, embodiments of the invention will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0006] Figure 1A An example of data stored in a container in a cloud storage system is shown;

[0007] Figure 1B An example of a data protection system that provides data protection operations in on-premises active tiering and cloud tiering, including deduplication and garbage collection, is shown;

[0008] Figure 2 An example of a container in a copy-and-forward cloud layer is shown;

[0009] Figure 3 An example of a cloud-based serverless architecture configured to perform garbage collection operations on a cloud tier of data stored in a cloud-based storage is shown;

[0010] Figure 4 Another example of a serverless architecture configured to perform garbage collection operations including copy-forwarding on a cloud tier of data is shown;

[0011] Figure 5 shows examples of how objects or files may be represented in cloud storage and how objects may be stored in containers;

[0012] Figure 6 An example of a method for performing garbage collection in cloud tiering is shown; and

[0013] Figure 7 A cost analysis is shown. DETAILED DESCRIPTION

[0014] Embodiments of the present invention generally relate to data protection, and to systems, methods, and devices for performing data protection operations. Examples of data protection operations include, but are not limited to, backup operations, restore operations, replication operations, disaster recovery operations, disaster recovery testing operations, failover or partial failover operations, data synchronization operations, replication operations, garbage collection operations, copy forwarding operations, etc., or combinations thereof. More particularly, embodiments of the present invention relate to systems, hardware, software, computer-readable media, and methods for performing data protection operations including garbage collection operations in a cloud.

[0015] Embodiments of the present invention relate to a data protection system (e.g., DELL EMC DATA DOMAIN) that includes or provides, for example, a local or active tier for data storage and a cloud tier for data storage. More particularly, embodiments of the present invention relate to garbage collection operations performed on the active tier and / or the cloud tier. In a deduplication storage system, data (objects, files, etc.) is typically divided into chunks or fragments. Garbage collection operations are performed to remove fragments that no longer correspond to data that should still be stored in the storage system. Unreferenced fragments are examples of dead fragments and can be removed by garbage collection operations, while live fragments should not be removed by garbage collection operations. Removing dead fragments can reduce storage requirements.

[0016] The data protection system can be an on-site appliance that provides active tiering of storage, and customers can use cloud tiering for certain data. Similar to on-premises data, data in the cloud is deduplicated and can be stored in containers that include compressed areas of fragments.

[0017] Figure 1A An example of data stored in cloud storage is shown. Figure 1A Cloud storage 150 is shown. The user's data can be stored in containers in the cloud. In this example, cloud storage 150 includes containers, which are represented as containers 152, 158, 160, 162, and 164. The number of containers is not limited, and the containers can be of different types. Some containers (such as containers 152, 158, 160, and 162) can store fragments in a compression area. Containers such as container 164 (CMETA container) can be metadata containers and can be configured to store metadata. Metadata can also be stored as fragments, and therefore, CMETA containers can also be cleaned up when metadata becomes dead.

[0018] For example, container 152 stores the fragments in compressed areas. The compressed areas of container 152 are represented as compressed areas 154 and 156. Each compressed area includes a fragment of data. Therefore, compressed areas 154 and 156 can each store a certain number of fragments. Compressed areas 154 and 156 can also be compressed and / or encrypted.

[0019] As described in more detail below, compression area 154 (similar to other compression areas) may contain both live and dead segments. In order to clean up container 152, for example, a copy of a compression area that is at least partially live (or includes any live segments) may be forwarded to a new container. Dead compression areas that do not contain any live segments are not copied and forwarded. Once the live compression area is copied and forwarded to the new container, the source container or the old container may be deleted. This reclaims storage space by removing the dead compression area from the container. For example, if compression area 154 is considered to be live and compression area 156 is considered to be dead, a garbage collection operation may copy compression area 154 to a new container and then delete container 152 to remove compression area 156 and the old version of compression area 154. These changes are reflected in CMETA container 164 (which may be cleaned up in a similar manner). This ensures that the data protection system knows the location of the segments and is able to perform deduplication.

[0020] By identifying specific compressed regions, the container 152 may be cleaned without having to decompress and / or decrypt the compressed regions or the segments stored therein.

[0021] Therefore, in order to free up storage space after data deletion, live compressed areas (including partially live compressed areas) are copied from the old container to the new container, and then the old container is deleted. More specifically, live compressed areas are copied to the new container (an example of copy-forwarding), and dead segments or dead compressed areas are deleted by deleting the old container. In this way, storage requirements can be reduced because the storage space previously used to store dead compressed areas is no longer needed. Therefore, dead segments are effectively removed.

[0022] In addition, when garbage collection operations are applied to compressed areas or to data ranges in containers, copy-forward operations can be performed without compressing and / or decrypting the compressed areas. In this case, the data protection system knows the starting address or offset of the compressed areas, and the starting address or offset is included in or can be derived from the locally stored metadata. Therefore, the live compressed areas can be copied and forwarded by identifying the source container, the starting address of the live compressed areas in the source container, the size of the compressed areas, and the destination container to which the live compressed areas are copied and forwarded.

[0023] Data deduplication can be managed by an on-premises appliance or on-premises data protection system, and embodiments of the present invention reduce costs by using instructions or schemes to enable larger data transfers or copy forwarding within the cloud itself to reduce reads and data transfers to the on-premises data protection system (transfers within the cloud do not incur costs). This significantly reduces the amount of data that would otherwise be transferred over the network to the on-premises system.

[0024] Embodiments of the present invention advance the functionality of the system and relate to a serverless architecture that can receive instructions from a field or on-premises data protection system. The data protection system maintains metadata that allows data to be deduplicated and allows identification and cleanup of live and / or dead compression zones in cloud tiers (and active tiers). This information included in or derived from the metadata allows instructions to be generated and sent to the cloud for execution. Instructions from the data protection system may include schemes that describe which compression zones to copy forward. These schemes can be applied to copy forward live compression zones from existing containers to new containers. This significantly eliminates network transmissions and significantly reduces customer costs, while supporting fine-grained cleanup of data stored in cloud tiers. More generally, embodiments of the present invention allow dead segments of an on-premises data protection system to be cleaned up by identifying data ranges that are considered to be live or partially live and then copying forwarding those ranges. In one example, this allows storage space to be reclaimed without decompressing / decrypting cloud objects. In one example, the range may or may not coincide with the boundaries of the compression zone.

[0025] The architecture for performing garbage collection operations includes a garbage collection engine of a data protection system (e.g., DATA DOMAIN), which in one example can access functions in a serverless cloud system (such as Amazon Lambda) by writing instructions to a memory or a specified URL (Uniform Resource Locator). Once an instruction is written to the specified URL or other events are detected, the function is instantiated and executed based on the instruction or information included in the instruction. Thus, the GC engine initiates a garbage collection operation, which is then performed by a function instance in the serverless architecture in the cloud tiering of the data.

[0026] Figure 1B An example of a system including or associated with a data protection system 110 (eg, DELL / EMC Data Domain) is shown. The data protection system 110 may be implemented, for example, as an appliance (physical or virtual) or using a server / agent configuration.

[0027] FIG. 1 shows a network 102. The network 102 represents an on-premises computing system and may be associated with servers / applications 108 (e.g., databases, email, file servers, virtualization). Clients 104 are able to use and access the servers and applications 108 through the infrastructure of the network 102. The network 102 is also associated with data 106 that may be stored on a storage device. The data 106 may be stored on the storage device and backed up by a data protection system 110 using a storage 112.

[0028] The data protection system 110 provides data protection services, which may include, but are not limited to, backup operations, restore operations, deduplication operations, garbage collection operations, etc. The data protection system may provide a memory 112 (also referred to as an active or local tier). Backups of the data 106 (and / or virtual machines, applications, objects (e.g., files) may be stored in the memory 112. In addition, the memory 112 may be deduplicated by the data protection system. The data protection system 110 may also include a garbage collection (GC) engine 114, which is configured to remove unreferenced objects or fragments from the memory 112.

[0029] The data protection system 110 may also provide or be associated with cloud storage (cloud 120). In one example, cloud 120 may be used to provide tiered storage 122 (also referred to as cloud tiering). The data protection system 110 may use tiered storage 122 to store some of the data stored on storage 112. For example, a customer may select certain files or data from storage 112 that should be migrated to and stored in cloud tiered storage 122. Thus, the data protection system 110 may write objects to cloud tiered storage 122, delete objects from cloud tiered storage 122, and perform other file or object-based operations on cloud tiered storage 122.

[0030] At the same time, the data protection system 110 may be responsible for deduplicating data at both the storage 112 and the cloud tiered storage 122. The data protection system 110 may also maintain metadata 116 that identifies live segments and / or dead segments in the storage 112 and / or storage 122 or allows for the identification of live segments and / or dead segments. More generally, the metadata 116 allows for the identification of live compression areas / dead compression areas. Thus, the data protection system 110 is also able to identify live or partially live data ranges and / or dead data ranges (e.g., dead compression areas / live compression areas). In one example, the identification of live segments effectively identifies the associated compression area as a live compression area. Once a compression area is determined to be a live compression area, it may not be necessary to repeat or evaluate all segments associated with the compression area.

[0031] The data protection system 110 is configured to minimize or reduce data transfer requirements to / from the cloud, at least in the context of garbage collection, and is configured to determine live compaction regions and is configured to clean up containers by removing or at least partially removing dead compaction regions, thereby removing dead segments, from the cloud tiered storage 122. Metadata 116 stored locally in the data protection system 110 allows the GC engine 114 to determine the liveness of segments or compaction regions during garbage collection operations and to make decisions about which data (e.g., extents, compaction regions) to forward from an existing cloud container copy to a new cloud container in the cloud tiered storage 122.

[0032] Figure 2 An example of a container stored in a cloud tier is shown and a copy-forward process is illustrated. Figure 2 Container 202 is shown. In this example, container 202 is an existing container that existed in the cloud before garbage collection operations began. Container 202 includes compressed regions (represented by compressed regions 204 and 208). Similar to the other compressed regions of container 202, compressed region 204 includes fragment 206. Compressed region 208 includes fragment 210.

[0033] At the start of a garbage collection operation, the GC engine 114 may determine that at least some of the segments 206 include live segments. The GC engine 114 may also determine that none of the segments 210 are live segments. In other words, the GC engine 114 may determine that the compressed area 208 is a dead compressed area, and the compressed area 204 is a live compressed area. In one example, a compressed area is live as long as it includes at least one live segment.

[0034] During a garbage collection operation, a new container 212 is created. The new container 212 will include a compressed area (represented by compressed area 214) when it is written. Similar to other compressed areas, compressed area 214 will include fragment 216 when it is written. During the copy-forward process, when writing to container 212, the live compressed areas of container 202 can be written to container 212. In this example, compressed area 204 is written to container 212 to become compressed area 214. Therefore, in one example, when a garbage collection operation is performed at the compressed area level, fragment 216 is the same as fragment 206. At this level, completely dead compressed areas such as compressed area 208 are deleted by the garbage collection operation. More specifically, compressed area 208 is deleted because compressed area 208 is not copied and forwarded. Then, container 202 is deleted.

[0035] During garbage collection, live compressed areas written into new container 212 may come from multiple source containers. In addition, metadata is created and stored as needed so that the new location of the compressed area, its contents, and the location of the fragments 216 are maintained by the data protection system.

[0036] When performing garbage collection operations at the compression area level, the compression area 204 is copied and forwarded by copying and forwarding based on the location of the compression area 204 within the container 202. In this example, the GC engine may forward the compressed area 204 to the container 202. Figure 1B In this example, the instructions (or other events) written to URL 126 may include the identifier of container 202, the starting position of compressed area 204 within container 202, and the size of compressed area 204. This allows function 124 (see Figure 1B) is instantiated to copy the compressed region 204 into the container 212. In addition, the compressed region 204 can be copied and forwarded without understanding the format of the compressed region 204 and without having to perform computationally expensive processes such as decompression and / or decryption. In other words, the compressed region 204 can be copied and forwarded while being compressed and / or encrypted.

[0037] Embodiments of the present invention allow for copy forwarding decisions without having to network transfer the data back to the source site (e.g., network 102). Network transfers can be avoided by creating functions that execute in response to events. These functions or instances have the advantage that they can read cloud containers and transfer data within the cloud without the transfer cost of reading back to the on-premises data protection system 110. When the garbage collection operation is complete, the function exits, so the computational cost remains low.

[0038] When scrubbing at the compression level, the recipe from the data protection system to write URL 126 may include a series of instructions such as:

[0039] Copy Length1 bytes from Start1 to Container C.

[0040] Copy the Start2 of Length2 bytes from container A to container C.

[0041] Copy Start3 of Length3 bytes from container B to container C.

[0042] Confirm that container C has the checksum value and write to the object storage.

[0043] Delete containers A and B.

[0044] Although the example scheme above focuses on copying byte ranges from the old container to the new container, these byte ranges may be selected to match compressed regions within the container.

[0045] As an integrity check, when the data protection system does not have the fragments themselves but only references to them, a checksum can be calculated on the fingerprint within the container. The fingerprint is stored in the metadata 116 associated with the container in the cloud. Metadata related to the fragments, compressed areas, and containers stored in the cloud can also be stored in the cloud.

[0046] Figure 3 Systems and methods are shown for performing garbage collection operations in the context of a system 300 (eg, an on-premises system) associated with an active tier and a cloud system 310 associated with a cloud tier. Figure 3A data protection system 302 (an example of the data protection system 110) is shown, which includes a GC engine 304 and can access metadata 306 associated with containers 320 stored in the cloud 310. The containers 320 can each include a compression area. However, some of the containers can store CMETA (container metadata). Therefore, the cloud storage 318 can also store other containers, such as CMETA containers.

[0047] During the garbage collection operation, the GC engine 304 can access the serverless function by writing a scheme 308 to a URL 324 (which can also be an API) or by performing another action that causes the execution of a function 312. Function 312 is configured to clean up a container 320. As shown, the data protection system 302 or the GC engine 304 instantiates the function 312 in the cloud by executing an event that calls function 312. Function 312 interacts with and cleans up the container 320 in the cloud storage 318 without incurring the data transfer cost as previously described. More specifically, GC instances 314 and 316 are specific instances of function 312. Each of instances 314 and 316 can execute different schemes. For example, each can work on different groups or ranges of containers.

[0048] The garbage collection operation is scalable and can act on a batch of containers. For example, containers can be processed in batches (the batch is also referred to as a series of containers). When processing a batch, the GC engine 308 can configure a scheme to allocate a portion of the batch to a function 312. The series of containers can be specified in the instruction or scheme 308. When multiple functions are instantiated, each function can act on a different portion of the batch. This allows the batch to be cleaned up in parallel. Multiple batches can be cleaned up continuously.

[0049] Figure 3 Also shown is a poll file 322. The poll file 322 can be created by instances 314 and 316 when garbage collection or the copy-forward aspect of garbage collection is completed. The GC engine 304 can periodically poll the cloud storage 318, and when the poll file 322 exists, the GC engine 304 understands that the function has been completed. In one example, the poll file 322 can include a checksum of the new container, which can be compared to a locally stored checksum to confirm the garbage collection operation.

[0050] Figure 4 An example of remote garbage collection using a serverless configuration (e.g., Amazon Lambda) is shown. The serverless configuration may include a function or multiple function instances that may be triggered by an event (e.g., such as by accessing a URL or API).

[0051] Figure 4A source site 420 (e.g., an on-premises site) is shown that includes a GC engine 402 as part of a data protection system. When the GC engine 402 is started, the GC engine 402 can start, configure, and deploy GC functions 404, 406, and 408 in a serverless layer 422 in the cloud. More specifically, the GC engine 402 can generate a recipe or instruction that includes sufficient information to clean up a container stored in a cloud object storage 424. When the recipe or instruction is written to a specified URL, for example, the functions 404, 406, and 408 are instantiated and executed on the container of the cloud object storage. The recipe can configure the operating parameters of the functions 404, 406, and 408.

[0052] Once functions 404, 406, and 408 are deployed and run, containers are cleaned up by copying forward live ranges specified in the instructions or scheme. When the instructions are at the compression region level, copy forward live compression regions (identified by offsets and sizes as previously discussed). In this example, live compression regions from containers C1 and C2 are copied into container C3, live compression regions from containers C4 and C5 are copied into container C6, and live compression regions from containers C7 and C8 are copied into container C9. Once copy forwarding is complete, containers C1, C2, C4, C5, C7, and C8 are deleted, and one or more polling files are created in cloud storage.

[0053] In one example, in a serverless configuration, the copy-and-forward method or process is encapsulated within a serverless framework. The data protection system then accesses the appropriate URL and supplies values ​​specifying the scheme to be applied.

[0054] The GC engine 402 may create a solution for each instance of the functions 404, 406, and 408 (e.g., each function handles a series of containers). The GC engine 402 creates the solution and writes the solution to the object store 424 or other specified URL. The specified URL may be the object store 424. This event triggers the functions 404, 406, and 408. When the event is detected, the solution may be parsed. When identified as a solution, the function corresponding to the event that may execute the solution is called.

[0055] When functions 404, 406, and 408 have completed, they can be configured to write a poll file or other object to object storage 424. In one example, a checksum can be written to object storage 424. GC engine 402 can periodically poll object storage 424 to check the checksum or other object. When an object or checksum exists, the checksum can be compared with the checksum stored locally as previously described. This can confirm that the copy-forward operation was performed correctly. In addition, a CMETA container is written at source site 420 and copied to cloud storage 424.

[0056] By way of example only, a container may have two parts: a metadata section, and a data section or compression section. In the data section, each compression section includes a bunch of compressed fragments. For each fragment, a fingerprint may be calculated for deduplication purposes. The metadata section represents or stores meta information of the container and may include various fields such as, but not limited to: how many compression sections there are in the container, the compressed and uncompressed sizes of the compression sections, the number of fragments in each compression section, the total number of fragments in the container, and a table of fingerprints of all fragments stored in the container.

[0057] Figure 5 An example of how a file or object can be represented in an active tier or in a cloud tier is shown. The storage system can represent an object or file as a fragment tree, such as Figure 5 500 . The content processing 502 may point to a super segment 504 (which may be an example of an L6 segment). The metadata segment 510 may include Lp segments (e.g., L1 to L6 segments). The data segment L0 508 is the actual data of the file or object and may be stored in a leaf node of the segment tree 500. The metadata segment 510 and the data segment 508 are usually stored in separate containers. Therefore, a container storing an Lp segment may be referred to as an Lp container, and a container storing an L0 segment may be referred to as an L0 container.

[0058] The CMETA container is a container for storing metadata. More specifically, the CMETA container can store a collection of metadata segments of the L0 container and the Lp container. During garbage collection, metadata (CMETA and Lp) is used to identify live segments and dead segments or to identify live compression areas and dead compression areas. In order to avoid reading metadata from the cloud, metadata (e.g., CMETA containers and Lp containers) are stored locally and copied to the cloud. In one embodiment, the L0 container is only written to the cloud. Therefore, in one embodiment, data is deduplicated with respect to both the active tier and the cloud tier. In one example, the CMETA container and the Lp container are maintained locally and in the cloud.

[0059] During garbage collection, the GC engine can read the local CMETA container. By reading the local CMETA container, the metadata of the actual L0 container and the Lp container can be obtained. The metadata may include fingerprints of all fragments of the actual L0 container and the Lp container.

[0060] Using this metadata, the GC engine can look up the fingerprint to determine the live and dead compression areas or ranges of the Lp container. The live compression areas or ranges of the existing Lp container are then copied to the new Lp container. The new Lp container is written locally and written to the cloud. Next, the metadata segments of the new Lp container are copied to the new CMETA container, and the new CMETA container is written locally and written to the cloud as an object.

[0061] Next, the metadata section of the local CMETA container is read to identify the live compressed region or range of the source or existing L0 container. This can be an iterative process. This allows the scheme (source_object_ID, offset, size) for the live compressed region in the existing container to be identified. These schemes are written to the URL so that a copy of the live compressed region can be forwarded to a new container or object in the cloud through a function in the serverless architecture.

[0062] Next, the metadata corresponding to the compressed region in the new container is copied locally to the new CMETA container and replicated to the cloud. Finally, the source Lp, L0, and CMETA containers (from which the region or range was copied) can be deleted to reclaim or release cloud space.

[0063] In a deduplication storage system, as previously discussed, containers can store different types of data. Some containers can store data fragments (e.g., L0 containers), while other containers can store data for storing and reconstructing objects (e.g., Lp containers). These containers can be similarly processed during garbage collection. In one example, these fragments are stored in a compression area.

[0064] In one example, the GC engine is aware of the containers storing data in the cloud tier. Figure 4 , the GC engine stores or has access to metadata describing containers C1, C2, C4, C5, C7, and C8. This allows the source container, live / dead compaction area to be identified from the metadata and included in one or more schemes.

[0065] In one example, a scheme may include a list of source containers from which live ranges are copied, the starting position and size of the ranges in the compression zone, a list of destination containers, and a list of destination CMETAs. Thus, the scheme identifies the containers to be cleaned, identifies the ranges of live data areas such as live compression zones, a list of destination containers, and the destination CMETA. The scheme is written to the appropriate location so that the function is called in response. Once this process is completed in the cloud tiering, a new CMETA container is written locally and in one example is copied to the cloud. The CMETA container corresponds to the cloud tiering container and may contain metadata related to the content stored in the cloud tiering. Thus, the CMETA container implements deduplication. The Lp container is also written locally.

[0066] When this process is complete and verified, the GC engine may poll the cloud object storage for the file to verify that the copy forward or garbage collection was completed and performed correctly.

[0067] By performing garbage collection in the cloud with a serverless architecture, the import / export cost is avoided. However, there is a computational cost to run these functions. Therefore, the cost of the serverless approach can be compared to the cost of transmitting these objects over the network (import / export cost). In one example, when reading 50 million objects and writing 25 million objects (each object is 4MB), the computational cost plus request cost plus get cost plus put cost can be about $422 per garbage collection cycle. This is generally less than the cost associated with transmitting these objects over the network.

[0068] Figure 6 An example of a method for performing data protection operations, such as garbage collection, on a cloud tier is shown. Figure 6 Methods performed by the data protection system as a whole are shown and aspects of garbage collection operations that may be performed by various components are shown. For example, GC engine 600 may be deployed on-premises and may be configured to control data protection operations, including garbage collection operations for active tiers and cloud tiers. Embodiments of the present invention may also be applicable to situations where the source is a cloud-based system.

[0069] The GC engine 600 may communicate with a serverless function 620 instantiated in the cloud in response to an event, such as a write by the GC engine 600. The serverless function 620 may be configured to manage garbage collection operations in the cloud by executing a recipe or controlling performance received from the GC engine 600. The serverless function 620 may perform a portion of the garbage collection operation including copy forwarding.

[0070] Figure 6 The illustrated method may begin by processing 602 metadata about cloud tiering. On-premises systems typically store metadata describing containers stored in the cloud, and processing the metadata may include evaluating the metadata of the containers stored in the cloud tiering to identify liveness and deadness of data stored in the containers. In one example, the metadata may allow the GC engine 600 to identify which compression regions include live segments and which compression regions contain only dead segments. As previously described, the metadata may also be used to identify live / dead compression regions.

[0071] Since the GC engine 600 knows the location of compressed regions and other aspects of the container, and since the state of the fragments and / or compressed regions can be determined, a scheme can be generated 604 so that live ranges or compressed regions can be copied and forwarded. At the compressed region level, these schemes allow compressed regions to be copied and forwarded regardless of format, compression, or encryption.

[0072] More specifically, in one example, the GC engine 600 knows how the data is compressed and / or encrypted. Therefore, the size of the range in its compressed / encrypted form can be determined. In one example, this information is present in the CMETA that describes the information stored in the container.

[0073] The plan generated by the GC engine 600 typically identifies the source container, the data location (e.g., offset or starting address in the container), the length or size, and the destination container. This information allows the function to copy data at a location corresponding to a live range or a compressed region to a new destination container.

[0074] Once generated, the scheme is written 606 to a specified location (a specific URL, such as an object cloud storage). The write is detected 622 and evaluated. When the write is interpreted as a specific event, the function corresponding to the detected event is instantiated 624. The scheme can be configured to call multiple functions or instances. Alternatively, multiple schemes can be sent so that each instantiated function operates on a different set of containers. Therefore, the launched function receives the scheme and performs 626 garbage collection operations according to the scheme.

[0075] Copy forwarding may be performed, for example, copying the range or compression region identified in the scheme from the source container to the new container. This may be performed for both the Lp container and the L0 container. The CMETA for the new container may be determined and stored in the new CMETA container. In one example, a new CMETA container is created in a similar manner.

[0076] In one example, the functions may each write a poll file to the object store, thereby signaling 628 completion of the garbage operation. These files may constitute a checksum that may be compared with a locally stored checksum available to the GC engine 600 to confirm that the garbage collection or copy-forward operation has been performed correctly. Thus, the GC engine 600 may poll 608 the object store until these files are detected.

[0077] Once the file is detected and verified, metadata about the cloud tiering may be updated 610. This may include receiving metadata associated with the new container. Metadata about the cloud tiering is updated 610 by the GC engine 600. This may include generating a new metadata container locally and copying the new metadata container to the cloud.

[0078] Figure 7 The cost of running a GC once a month at a cloud provider is shown. This cloud provider has two storage configurations: Standard and Standard Infrequent Access. Other cloud providers have similar costs.

[0079] In this example, a cloud tier of 1PB is assumed. Based on experience, it is determined that there may be 10% loss per GC cycle. Therefore, in 1PB, it is expected that 100TB of data will be dead. Due to segmentation, live data and dead data will be distributed in containers (i.e., containers will have both live and dead data). The average viability of the cleaned container is 50% (i.e., 50% of the fragments are dead and 50% are alive). Based on these assumptions, in order to clean up 100TB of dead data with an average viability of 50%, the GC needs to read 200TB of data from the cloud and write 100TB of new data to the cloud. If each object size is 4MB, the GC needs to read 200TB / 4MB=52428800 objects and write 52428800 / 2=26214400 objects to the cloud. The transaction cost of the put (PUT) operation is $0.005 per 1,000 requests, and the get cost is $0.0004 per 1,000 requests. So the transaction cost of reading 52428800 objects is $262 and writing is $10. But the main cost is the data transfer cost: import / export cost. The data transfer cost of reading 200TB and writing 100TB is $14,994 for standard storage and $7826 for infrequent access.

[0080] like Figure 7 As shown, the cost savings from the cleaned data is $2,126.25 because the monthly data storage cost on the cloud is reduced.

[0081] Check the computational costs associated with a serverless configuration. In one example, the configuration may have 16 cores, 64GB of memory and cost $0.80 per hour. In addition, the speed of this instance may also vary. In one example, a copy speed of at least 200MB / s may be achieved. If 8 instances are started, a speed of 1.6GB / s may be achieved. In this example and at this speed, copying and forwarding 200TB of data will take about 12,800 seconds or about 35 hours. This results in a cost of $227.

[0082] Thus, embodiments of the present invention achieve a cost reduction of approximately 66x per GC cycle. Furthermore, this is faster because as many instances can be started as desired or the number can be controlled based on the data to be copied. In contrast, copy-forwarding over the network is significantly more expensive and slower. Assuming the network has 100MB bandwidth, 100MB / s reads cannot be achieved in the best case scenario. Thus, embodiments of the present invention save network bandwidth for moving data to the cloud, rather than wasting it on conventional garbage collection methods.

[0083] The following is a discussion of aspects of an example operating environment for various embodiments of the invention.The following discussion is not intended to limit the scope of the invention or applicability of the embodiments in any way.

[0084] In general, embodiments of the present invention may be implemented in conjunction with systems, software, and components that individually and / or collectively implement and / or cause data management operations to be implemented. Such data management operations may include, but are not limited to, data read / write / delete operations, data backup operations, data recovery operations, data cloning operations, data archiving operations, and disaster recovery operations. Therefore, although in some aspects, the discussion herein may involve discussions of data protection environments and operations, the scope of the present invention is not limited thereto. More generally, then, the scope of the present invention encompasses any operating environment in which the disclosed concepts may be useful. By way of illustration and not limitation, embodiments of the present invention may be used in conjunction with data backup and recovery platforms such as Dell-EMC NetWorker and Avamar and Data Domain platforms.

[0085] The data protection environment may take the form of a public or private cloud storage environment, an on-premises storage environment, and a hybrid storage environment including public and private elements, but the scope of the present invention also extends to any other type of data protection environment. Any of these example storage environments may be partially or completely virtualized. The storage environment may include or consist of a data center that is operable to serve read and write operations initiated by one or more clients.

[0086] In addition to the storage environment, the operating environment may also include one or more host devices, such as clients, that each host one or more applications. Thus, a particular client may employ one or more instances of each of the one or more applications or otherwise be associated with the one or more instances. In general, the applications employed by the client are not limited to any particular function or any particular type of function. For example, some example applications and data include email applications (such as MS Exchange), file systems, and databases (such as Oracle databases and SQL server databases). The application on the client may generate new data and / or modified data that is desired to be protected.

[0087] Any of the devices or entities disclosed herein may be protected by one or more data protection policies according to various embodiments of the present invention. Still other examples of devices that may be protected by data protection policies according to embodiments of the present invention include, but are not limited to, containers and VMs.

[0088] Any device in the operating environment (including clients, servers, and hosts) may take the form of software, physical machines, or virtual machines (VMs), or any combination of these, but any implementation does not require a specific device implementation or configuration. Similarly, data protection system components (e.g., such as databases, storage servers, storage volumes (LUNs), storage disks, copy services, backup servers, recovery servers, backup clients, and recovery clients) may also take the form of software, physical machines, or virtual machines (VMs), but any implementation does not require a specific component implementation. In the case of using VMs, a hypervisor or other virtual machine monitor (VMM) may be used to create and control the VMs.

[0089] As used herein, the scope of the term "data" is intended to be broad. Thus, the term encompasses, by way of example and not limitation, data segments, data chunks, data blocks, atomic data, emails, objects of any type, files, contacts, directories, subdirectories, volumes, containers, and any combination of one or more of the above, such as may be produced by a data stream segmentation process.

[0090] Example embodiments of the present invention are applicable to any system capable of storing and processing various types of objects in analog, digital or other forms. Although terms such as documents, files, blocks or objects may be used by way of example, the principles of the present invention are not limited to any particular form of representing and storing data or other information. Rather, such principles are equally applicable to any object capable of representing information.

[0091] The embodiments disclosed herein may include the use of a special or general purpose computer including various computer hardware or software modules, as discussed in more detail below. The computer may include a processor and a computer storage medium carrying instructions that, when executed by the processor and / or caused to be executed by the processor, perform any one or more of the methods disclosed herein.

[0092] As indicated above, embodiments within the scope of the present invention also include computer storage media, which are physical media for carrying or storing computer-executable instructions or data structures. Such computer storage media can be any available physical media that can be accessed by a general or special purpose computer.

[0093] By way of example and not limitation, such computer storage media may include hardware memory, such as a solid state disk / device (SSD), RAM, ROM, EEPROM, CD-ROM, flash memory, phase change memory ("PCM"), or other optical disk storage, magnetic disk storage or other magnetic storage, or any other hardware storage that can be used to store program code in the form of computer-executable instructions or data structures that can be accessed and executed by a general or special purpose computer system to implement the functionality disclosed herein. Combinations of the foregoing should also be included within the scope of computer storage media. Such media are also examples of non-transitory storage media, and non-transitory storage media also encompass cloud-based storage systems and structures, but the scope of the present invention is not limited to these examples of non-transitory storage media.

[0094] Computer executable instructions include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a special-purpose processing device to perform a specific function or group of functions. Although the subject matter has been described in language specific to structural features and / or method actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions disclosed herein are disclosed as example forms of implementing the claims.

[0095] As used herein, the term "module" or "component" may refer to a software object or routine that executes on a computing system. The various components, modules, engines, and services disclosed herein may be implemented as objects or processes that execute on a computing system, for example, as separate threads. Although the systems and methods described herein may be implemented in software, it is also possible and conceivable to implement them in hardware, or a combination of software and hardware. In the present invention, a "computing entity" may be any computing system as previously described herein or any module or combination of modules that runs on a computing system.

[0096] In at least some cases, a hardware processor is provided that is operable to execute executable instructions to perform a method or process, such as the methods and processes disclosed herein. The hardware processor may or may not include elements of other hardware, such as the computing devices and systems disclosed herein.

[0097] In terms of computing environment, embodiments of the present invention may be executed in a client-server environment (whether a network environment or a local environment) or in any other suitable environment. Suitable operating environments for at least some embodiments of the present invention include cloud computing environments in which one or more of a client, server, or other machine may reside and operate in a cloud environment.

[0098] The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics of the present invention. The described embodiments should be considered in all respects as illustrative only and not restrictive. Therefore, the scope of the present invention is indicated by the appended claims rather than the foregoing description. All changes within the equivalent meaning and scope of the claims will be included within their scope.

Claims

1. A method for performing a garbage collection operation in a computing system, the computing system providing an active tier for data at a field device, and a cloud tier for data, the method being performed in the cloud tier, the cloud tier including a cloud storage of a serverless cloud system, the method comprising: processing, by a garbage collection engine, metadata about the cloud tier stored in the active tier to identify containers in the cloud storage to be cleaned in the cloud tier, wherein the identified containers include regions, some of which include both dead segments and live segments; generating, by the garbage collection engine, a scheme identifying a location of the live fragment in the identified region of the identified container based on metadata identifying the live fragment and the dead fragment stored in the active hierarchy, wherein the scheme specifies a byte range of the identified region of the identified container; Writing the scheme into a uniform resource locator; detecting an event that the scheme has been written to the uniform resource locator; invoking a function in the serverless cloud system in response to detecting the event; as well as The scheme is executed by the function, wherein the function copies the live fragment from the identified container to a new area in the new container by copying the data within the byte range to the new container and then deleting the identified container, wherein the live fragment is copied and forwarded without considering the format, compression status, and encryption status.

2. The method of claim 1, wherein: The region includes a compressed region, wherein each scheme identifies at least a container, a location of the compressed region in the container, a size of the compressed region, and a destination container for storing the live segment, wherein the live segment is stored in a new compressed region in the destination container.

3. The method of claim 1, wherein: The scheme is executed by a plurality of the functions, so that cloud tiers of data are cleaned in parallel.

4. The method of claim 1, further comprising updating the metadata stored in the active tier to reflect the location of the new region in the new container stored in the cloud tier after executing the solution.

5. The method of claim 1 , further comprising identifying metadata of an L0 container and an Lp container stored in the cloud storage from the metadata about the cloud tiering, the metadata of the L0 container and the Lp container comprising fingerprints of fragments in the L0 container and the Lp container.

6. The method of claim 5, further comprising performing a lookup to identify live and dead regions of the Lp container.

7. The method of claim 6, further comprising generating the scheme that allows copying the live area from the Lp container into a new Lp container.

8. The method of claim 7, further comprising writing the new Lp container locally and writing the new Lp container to the cloud tier.

9. The method of claim 8, further comprising copying metadata of the new Lp container to a new CMETA container, wherein the new CMETA container is written locally and the new CMETA container is written to the cloud tier.

10. The method of claim 1, wherein: The container includes a local CMETA container, an Lp container, and an L0 container, and the method further includes: Repeating the metadata section of the local CMETA container to identify the active area of ​​the L0 container; forming the solution based on the local CMETA container; locally copying the metadata corresponding to the schema into a new CMETA container and replicating the new CMETA container to the cloud tier; and The Lp container, the L0 container, and the CMETA container from which the live area is forwarded are deleted to reclaim space in the cloud storage.

11. The method of claim 1, wherein: The uniform resource locator includes a uniform resource locator of the cloud storage.

12. The method of claim 1, further comprising polling the cloud storage for a polling file written by the function, wherein the polling file allows a data protection system to verify the scheme executed by the function.

13. The method of claim 12, wherein: The poll file includes a checksum, which is compared to a locally stored checksum.

14. A non-transitory computer readable medium comprising computer executable instructions that, when executed, perform garbage collection operations in a cloud tier of data, wherein the cloud tier of data is associated with a computing system, the computing system also having an active tier of data at a field device, the cloud tier comprising cloud storage of a serverless cloud system, the operations comprising: processing, by a garbage collection engine of a data protection system of the field device, metadata about the cloud tier stored in the active tier to identify, in the cloud tier, containers in the cloud storage to be cleaned, wherein the identified containers contain regions, some of which include both dead segments and live segments; generating, by the garbage collection engine, a scheme identifying a location of the live fragment in the identified region of the identified container based on metadata identifying the live fragment and the dead fragment stored in the active hierarchy, wherein the scheme specifies a byte range of the identified region of the identified container; Writing the scheme into a uniform resource locator; detecting an event that the scheme has been written to the uniform resource locator; invoking a function in the serverless cloud system in response to detecting the event; as well as The scheme is executed by the function, wherein the function copies the live fragment from the identified container to a new area in the new container by copying the data within the byte range to the new container and then deleting the identified container, wherein the live fragment is copied and forwarded without considering the format, compression status, and encryption status.

15. The non-transitory computer readable medium of claim 14, wherein: The region includes a compressed region, wherein each of the schemes identifies at least an existing container, a location of a compressed region, a size of the compressed region, and a destination container for storing the live segment, wherein the live segment is stored in a new compressed region in the destination container.

16. The non-transitory computer readable medium of claim 14, wherein: The uniform resource locator includes a uniform resource locator of the cloud storage.

17. The non-transitory computer-readable medium of claim 14, further comprising polling the cloud storage for a polling file written by the function, wherein the polling file allows the data protection system to verify the scheme executed by the function.

18. The non-transitory computer readable medium of claim 14, further comprising: identifying metadata of an L0 container and an Lp container stored in the cloud storage from the metadata, the metadata of the L0 container and the Lp container comprising fingerprints of segments in the L0 container and the Lp container; performing a lookup to identify live and dead segments of the Lp container; generating said scheme allowing copying said live segments from said Lp container into a new Lp container; Writing the new Lp container locally and writing the new Lp container to the cloud storage; Copying metadata of the new Lp container to a new CMETA container, wherein the new CMETA container is written locally and the new CMETA container is written to the cloud storage; The metadata section of the local CMETA container is repeated to identify the live compression area of ​​the L0 container; forming the solution based on the local CMETA container; as well as The metadata corresponding to the schema is copied locally into a new CMETA container and the new CMETA container is replicated to the cloud storage.

Images