Preventing resource scarcity in backup sessions

Abstract

A method for performing a backup operation, the method comprising: receiving a backup request for an asset; segmenting the plurality of slices of the asset into a plurality of backup sessions based on available capacity in a proxy infrastructure, where each backup session includes a separate portion of the slice, where the plurality of slices are associated with a queue; sending a first reservation request to the proxy infrastructure, wherein the first reservation request specifies the plurality of backup sessions; making a first determination that a first portion of the plurality of backup sessions is successfully reserved and a second portion of the plurality of backup sessions is not successfully reserved; sending a request to initiate the first portion of the plurality of backup sessions on the proxy infrastructure; and sending the second portion of the plurality of backup sessions to the queue based on the first determination.

Description

Background Art

[0001] A computing device may include any number of internal components, such as a processor, memory, and persistent storage. Computing resources associated with (e.g., used by) each of these internal components may be used to generate, store, and back up data. Such utilization of computing resources may affect the overall performance of the computing device and the ability to adequately back up data. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Certain embodiments of the present technology will be described with reference to the accompanying drawings. However, the drawings only illustrate certain aspects or implementations of the present technology by way of example and are not meant to limit the scope of the claims.

[0003] Figure 1 A diagram of a system in accordance with one or more embodiments of the present technology is shown.

[0004] Figure 2 A diagram of a data manager in accordance with one or more embodiments of the present technology is shown.

[0005] Figure 3.1 A diagram of a proxy infrastructure is shown in accordance with one or more embodiments of the present technology.

[0006] Figure 3.2 A diagram of a proxy host is shown in accordance with one or more embodiments of the present technology.

[0007] Figure 4.1 to Figure 4.3 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown.

[0008] Figure 5.1 to Figure 5.4 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown.

[0009] Figure 6.1 to Figure 6.2 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown.

[0010] Figure 7 A computing system in accordance with one or more embodiments of the present technology is shown. DETAILED DESCRIPTION

[0011] In general, embodiments of the present technology relate to efficiently using backup system resources when backing up assets. In the context of the present technology, an asset is data or metadata associated with a file, folder, and / or any other data hierarchy (or structure) (e.g., a volume). More specifically, embodiments relate to determining how to efficiently create and process backup sessions on a proxy infrastructure (i.e., the infrastructure of a backup system used to generate backups).

[0012] refer to Figure 4.1 to Figure 4.3 In one embodiment of the present technology, the agent manager (see e.g. Figure 3.1 ) uses a reservation mechanism to optimize the creation of backup sessions. In this way, only the number of backup sessions that the proxy infrastructure can actually handle when a reservation request occurs is created, and therefore, no (or minimal) resources are wasted in creating backup sessions. In addition, once backup sessions are created, they are likely to be successfully processed because the proxy infrastructure (e.g., Figure 1 ) to process the backup sessions. This minimizes backup sessions that are created but not processed within the backup window.

[0013] Reference now Figure 4.1 and Figure 5.1 to Figure 5.4 In one embodiment of the present technology, the asset is divided into slices. Then, the proxy manager creates a backup session (with Figure 4.2 to Figure 4.3 In contrast, there are multiple slices per backup session), and a subset of backup sessions are issued to the proxy infrastructure. Each of these backup sessions is then executed in parallel in its own container. Once a backup session in a given container is completed, the proxy manager is notified and another backup session is issued to the proxy infrastructure. In this way, the proxy manager is able to maintain the processing of backup sessions at an optimal level because the proxy manager is able to manage the workload on the proxy infrastructure on a per-slice basis.

[0014] Reference now Figure 4.1 and Figure 6.1 to Figure 6.2 In one embodiment of the present technology, the proxy manager uses a reservation mechanism to optimize the creation of backup sessions, where each backup session includes a set of slices. In this way, only the number of backup sessions that the proxy infrastructure can actually handle when a reservation request occurs is created, and no (or minimal) resources are wasted when creating backup sessions. In addition, once the backup sessions are created, they are likely to be successfully processed because there should be enough resources on the proxy infrastructure to process the backup sessions.

[0015] In this embodiment, a container is instantiated in the proxy infrastructure for each backup session. When a slice in a session has been processed, the proxy infrastructure is notified and another slice is added to the backup session. In this way, the number of backup sessions being executed remains relatively constant, and backup sessions can continue to execute while there are still unprocessed slices in the asset. In addition, this embodiment enables the continued use of existing containers (rather than dismounting / terminating existing containers for completed backup sessions and creating new containers for each new backup session, see e.g. Figure 4.2 to Figure 4.3 and Figure 5.1 to Figure 5.4 ).

[0016] The above three methods can be used together or separately, and can be used in Figure 1 to Figure 3.2 is implemented on the system shown.

[0017] One or more embodiments are described below.

[0018] Figure 1 A diagram of a system according to one or more embodiments described herein is shown. The system (100) may include a data manager (102), a proxy infrastructure (104), one or more primary storage devices (106), one or more backup storage systems (108), and a network attached storage device (NAS) (110). Each of these components is described below.

[0019] In one or more embodiments, Figure 1 The components shown may be operatively connected to each other via a network (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, etc.). The network may be implemented using any combination of wired and / or wireless connections. In addition, the network may encompass various interconnected network-enabled subcomponents (or systems) (e.g., switches, routers, gateways, etc.) that may facilitate communication between the above-mentioned system (100) components. In addition, the components may communicate with each other using any combination of wired and / or wireless communication protocols.

[0020] In one or more embodiments, the data manager (102) is configured to manage the backup of assets from the primary storage device (106) to the backup storage device (108). The data manager (102) is operatively connected to a network attached storage device (NAS) (110), which is Figure 2 The data manager (102) includes a program for executing Figure 4.1 to Figure 6.2 (described below) The functionality of some or all of the methods shown.

[0021] In one or more embodiments, the data manager (102) may be implemented as a physical computing device or a logical computing device. In one or more embodiments, the physical computing device may be, for example, a laptop computer, a desktop computer, a server, a distributed computing system, or a cloud resource. The physical computing device may include one or more processors, a memory (e.g., RAM), and a persistent storage device (e.g., a disk drive, an SSD, etc.). The physical computing device may include instructions stored in the persistent storage device that, when executed by a processor of the physical computing device, causes the physical computing device to perform the functionality described throughout this application. In one or more embodiments, the logical computing device may utilize the computing resources of any number of computing devices to provide the functionality of the data manager described throughout this application.

[0022] exist Figure 2 Additional details about the data manager are provided in .

[0023] In one or more embodiments, the proxy infrastructure (104) may represent the physical components of the system that assist in instantiating a proxy host for secure backup operations. The proxy infrastructure (104) acts as an intermediary between the primary storage device (106) and the backup storage device (108) when it receives a backup request from the data manager (102). The proxy infrastructure (104) may provide one or more functionalities such as (but not limited to): load balancing (to evenly distribute traffic among different proxy hosts to prevent any single proxy host from becoming overloaded), controlling and managing proxy hosts, and removing containers during backup sessions. The proxy infrastructure (104) is software intended to orchestrate NAS asset protection using multiple NAS proxy hosts (304A-304N) and will be discussed in detail below. The proxy infrastructure (104) may include software for performing Figure 4.1 to Figure 6.2 The functionality of some or all of the methods shown.

[0024] exist Figure 3.1 to Figure 3.2 Additional details about the proxy infrastructure are provided in .

[0025] In one or more embodiments, the primary storage device (106) may represent a hardware component for storing data. In one embodiment, the primary storage device (106) may be configured as a storage device array (e.g., a network attached storage device (NAS) array), where a storage device array may refer to a collection of one or more physical storage devices. Each physical storage device may include a non-transitory computer-readable storage medium in which data may be stored in whole or in part and temporarily or permanently.

[0026] In one or more embodiments, the backup storage system (108) may represent a data backup, archiving, and / or disaster recovery storage system. The backup storage system (108) may be implemented using one or more servers (not shown). Each server may refer to a physical server or a virtual server that may reside in a cloud computing environment. Additionally or alternatively, the backup storage system (108) may use a server similar to Figure 7 The exemplary computing systems shown may be implemented by one or more computing systems.

[0027] The backup storage system (108) may include a Figure 4.1 to Figure 6.2 (described below) The functionality of some or all of the methods shown.

[0028] Although Figure 1 A configuration of components is shown, but other system (100) configurations may be used without departing from the scope of the present technology. Figure 1The components in are shown as distinct components, but one or more of these components may be co-located on the same physical hardware without departing from the present technology.

[0029] Now go to Figure 2 , illustrates a data manager according to one or more embodiments of the present technology. The data manager (200) includes a network attached storage device (NAS) data manager (202) and is operatively connected to the NAS storage (204). Each of the above components is described below.

[0030] In one or more embodiments, the NAS data manager (202) may be implemented in hardware, software, or any combination thereof. The NAS data manager (202) includes functionality for managing the overall backup of assets from primary storage to a backup storage system. This functionality may include executing Figure 4.1 to Figure 6.2 Some or all of the steps shown.

[0031] In one or more embodiments, the NAS (204) includes volatile and non-volatile (i.e., persistent) storage. The NAS (204) is used by the NAS data manager to temporarily store slices in per-asset queues (206) (described below). Each of the asset queues (e.g., 206A, 206N) is populated with slices associated with the asset being backed up. The slices are maintained in the above-mentioned asset queues (206A, 206N) until they are processed in the backup session. See, e.g., Figure 4.1 to Figure 6.2 .

[0032] Although Figure 2 A configuration of components is shown, but other data manager (200) configurations may be used without departing from the scope of the present technology.

[0033] Now go to Figure 3.1 , Figure 3.1 A proxy infrastructure according to one or more embodiments of the present technology is shown. The proxy infrastructure (300) may include a proxy manager (302) and one or more proxy hosts (304A-304N). Each of these proxy infrastructure components is described below.

[0034] In one or more embodiments, the agent manager (302) can be implemented using hardware, software, or any combination thereof. In one embodiment, the agent manager (302) can be implemented using a physical computing device (described above) or a logical computing device (described above).

[0035] Proxy Manager ( Figure 2 , 200) and the data manager ( Figure 2, 200) to facilitate the backup of assets. More specifically, the proxy manager (302) includes the following functionality: receiving a request to process a backup session (which includes slices), and in response to receiving the request, allocating appropriate physical or logical infrastructure (also referred to as a proxy host (e.g., 304A, 304N)) to process the backup session (i.e., transfer the slices in the backup session to the backup storage system). To perform the above functionality, the proxy manager (302) may execute Figure 4.1 to Figure 6.2 at least some of the steps in .

[0036] In one or more embodiments, the proxy host (e.g., 304A, 304N) can be implemented using hardware, software, or any combination thereof, and includes functionality to handle backup sessions. In one embodiment, the proxy host (e.g., 304A, 304N) can be implemented using a physical computing device (described above) or a logical computing device (described above).

[0037] Although Figure 3.1 A configuration of components is shown, but other proxy infrastructure (300) configurations may be used without departing from the scope of the present technology.

[0038] Figure 3.2 A proxy host according to one or more embodiments of the present technology is shown. The proxy host (304) may include one or more containers (e.g., 320A, 320N). Each container may include a container proxy (e.g., 322A, 322N) and a container storage device (e.g., 324A, 324N). Each of these components is described below.

[0039] In one or more embodiments, each container (e.g., 320A, 320N) is software that executes on an agent host (e.g., 304) and is managed (directly or indirectly) by an agent manager (e.g., 302). In one or more embodiments where a container executes as an isolated software instance, the container may establish a semi-isolated virtual environment in which one or more applications (e.g., container agents (e.g., 322A, 322N)) execute. The container may execute an agent host (304) operating system (e.g., a software layer that utilizes low-level system components to execute software) in "user space". As a non-limiting example, a container may be container.

[0040] In one or more embodiments, the container agent (e.g., 322A, 322N) is software executed in the container that includes functionality to process the backup session and communicate the processing status of the backup session to the agent manager. The container agent may also execute Figure 4.1 to Figure 6.2 Zero, one, or more of the steps shown.

[0041] In one or more embodiments, a container storage device (e.g., 324A, 324N) is a storage device (which may be volatile, non-volatile storage device, or any combination thereof) that stores slices that need to be processed by a container agent (e.g., 322A, 322N).

[0042] Although the container agent has been described as performing a variety of different functions, a container may include multiple different applications (or agents); each application performing one or more of the above functions without departing from the present technology.

[0043] Although Figure 3.2 A configuration of components is shown, but other proxy host (304) configurations may be used without departing from the scope of the present technology.

[0044] Hosting reservation

[0045] Traditionally, backup sessions are created and provided to a proxy host without any knowledge of the resources available on the proxy host to actually process the backup session within the backup window. Therefore, if the proxy host is resource constrained, one or more backup sessions will not be processed and the entire backup will fail. In the just-in-time approach, a reservation mechanism is used to obtain information about the current resources available on the proxy host. This information is then used to optimize the number of backup sessions that are created and submitted to the proxy host. In this way, the proxy host resources are better utilized and the likelihood that a backup session provided to the proxy host will not be completed is reduced.

[0046] Figure 4.1 to Figure 4.3 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown. Figure 4.1 to Figure 4.3 The method shown may be performed by, for example, a NAS data manager (202). Without departing from the scope of the present invention, Figure 1 to Figure 3.2 Other components of the system can execute Figure 4.1 to Figure 4.3 All or part of the method.

[0047] All or part of the method shown may be performed by one or more components of the system. Although the various steps in this flowchart are presented and described sequentially, it should be understood by those skilled in the art that some or all of the steps may be performed in a different order, combined or omitted, and some or all of the steps may be performed in parallel.

[0048] refer to Figure 4.1 , which shows a method for generating slices for assets to be backed up. These slices are then Figure 4.2 to Figure 4.3 , Figure 5.1 to Figure 5.4 and Figure 6.1 to Figure 6.2 to handle.

[0049] Go to Figure 4.1 In step 400, a request to back up an asset is received. In one embodiment, the request may be for a single asset (eg, a file, a folder, a database, a volume, etc.) or a group of assets.

[0050] In step 402, a snapshot of the asset is obtained in response to the request in step 400. A snapshot corresponds to a copy of an asset at a given time. In one embodiment, the snapshot may be generated on the primary storage device in response to a request from a NAS data manager.

[0051] In step 404, one or more slices of the asset are generated from the snapshot. More specifically, the snapshot is divided into a plurality of parts, where each part of the snapshot is a slice and is associated with a slice identifier (ID). The size of a given slice (or slice group) may vary based on the implementation of the technology.

[0052] In step 406, at NAS ( Figure 2 , 204) on an asset queue (see e.g. Figure 2 , 206A). Once generated, the asset queue is populated with the slice IDs corresponding to the slices generated in step 404.

[0053] Figure 4.2 to Figure 4.3 Describes the process used to Figure 4.1 Methods for generating slices in . Figure 4.2 , in step 410, a populated asset queue is selected. If an asset queue includes at least one slice ID, it is considered populated. The selection of a given populated asset queue may be performed using any known or later discovered method. For example, the selection may be random, may be based on the number of slice IDs remaining in each populated asset queue, may be based on the relative importance of assets associated with each populated asset queue. The present technology is not limited to the above examples.

[0054] A capacity request is sent to the proxy infrastructure (or more specifically the proxy manager) in step 412. The capacity request is used to determine the resource availability of the proxy infrastructure for processing the slice associated with the slice ID in the selected populated asset queue.

[0055] In step 414, a capacity response is received from the proxy infrastructure, the capacity response specifying resource availability of the proxy infrastructure. In one embodiment, the capacity response may specify the number of backup sessions that the proxy infrastructure can handle (e.g., it can handle in parallel) based on its current resource availability. Other measures of resource availability may be specified in the capacity response without departing from the present technology.

[0056] In step 416, slices (or more specifically, slice IDs) from the selected populated asset queue are assigned to one or more backup sessions, where each backup session may include at most the specified number of slice IDs, and where the number of backup sessions does not exceed the resource availability specified in the capacity response. For example, consider the following scenario: the capacity response specifies four backup sessions and each backup session includes at most eight slice IDs. In this scenario, four backup sessions will be created, each specifying eight slice IDs.

[0057] In step 418, a reservation request is sent to the proxy infrastructure. This reservation request specifies the set of backup sessions generated in step 416. Continuing with the example in step 416, the reservation request specifies four backup sessions.

[0058] In step 420, a reservation response is received from the proxy infrastructure. The reservation response specifies that the proxy infrastructure is not able to process any of the backup sessions specified in the reservation request, is able to process some or all of the backup sessions. The reservation response may specify (i) backup sessions that can be processed (i.e., successfully reserved) and / or (ii) backup sessions that cannot be processed (i.e., unsuccessfully reserved).

[0059] In step 422, a determination is made as to whether all backup sessions in the reservation request are successfully reserved (i.e., the proxy infrastructure is able to process all backup sessions specified in the reservation request). The determination in step 422 is based on the reservation response. If all backup sessions in the reservation request are not successfully reserved, the method proceeds to step 424; otherwise, the method proceeds to step 426.

[0060] In step 424, based on the determination made in step 422, if the entire portion of the backup session in the reservation request is not successfully reserved, the slice ID associated with the unsuccessfully reserved backup session is placed at the end of the corresponding asset queue.

[0061] In step 426, based on the determination made in step 422, if the entire portion of the backup sessions in the reservation request was successful, processing of all successfully scheduled backup sessions is initiated on the proxy infrastructure. In one embodiment, each backup session is associated with a proxy host (e.g., Figure 3.2 , 304) on a container (e.g., Figure 3.2 , 320A). The container includes the following functionality: (i) selecting a slice ID from the backup session, (ii) reading a slice associated with the slice ID from the primary storage device, and (iii) writing the slice to an appropriate location in the backup storage system.

[0062] Once all slice IDs in a given session have been processed (i.e., each slice corresponding to a slice ID specified in the backup session has been read from primary storage and written to the backup storage system), the container agent ( Figure 3.2 , 322A) to the agent manager ( Figure 3.1 , 302). Then, the proxy manager sends a notification to the NAS data manager ( Figure 2 , 202) issue a notice.

[0063] In one embodiment, once a container has finished processing a backup session, the container is torn down (or otherwise terminated) on the proxy infrastructure to free up resources for other containers to be instantiated.

[0064] Reference now Figure 4.3 , Figure 4.3 How the NAS data manager processes the above notification from the proxy manager is described. In step 430, the NAS data manager receives a notification from the proxy infrastructure that a backup session on a container on the proxy host has been fully processed.

[0065] In step 432, a determination is made as to whether there are any remaining filled asset queues. If there are any remaining filled asset queues, the process proceeds to step 410 (at Figure 4.2 otherwise, the process ends.

[0066] Each time a backup session is fully processed, the Data Manager can execute Figure 4.2 to Figure 4.3 Thus, at any given time, the proxy infrastructure may be processing backup sessions associated with different assets. Figure 4.1 The method in Figure 4.2 to Figure 4.3 The methods shown are performed simultaneously.

[0067] Non-limiting examples intended to highlight various aspects of the present technology are described below.

[0068] Continuing with this example, consider the following scenario: there are ten assets, and each of these assets is divided into 80 slices, for a total of 800 slices for all ten assets. Once the slices are created, the slice IDs are placed in the data manager (e.g. Figure 2 , 200). Therefore, each of the per-asset queues includes 80 slice IDs corresponding to the slices of the assets associated with the per-asset queue.

[0069] At a later point in time, the NAS data manager (e.g., Figure 2 , 202) to the proxy infrastructure (e.g., Figure 1, 104) sends a capacity request. In this scenario, the proxy infrastructure sends a capacity response indicating that it has capacity for five backup sessions, each with up to eight slices. The NAS data manager selects an asset and then obtains 40 slice IDs from the corresponding asset queue to generate five backup sessions, each with eight slices.

[0070] The NAS data manager then sends a reservation request to the proxy infrastructure, where the reservation request specifies five backup sessions. After receiving the reservation request, the proxy infrastructure attempts to schedule backups on the proxy host (e.g., Figure 3.1 , 304A, 304N) create five containers (one container for each backup session) (e.g., Figure 3.2 , 302A, 320N). However, in this scenario, the proxy infrastructure has less capacity than previously indicated when it sent the above capacity response. Therefore, the proxy infrastructure is only able to create four containers, and thus can only process four of the five backup sessions. Therefore, the slice ID associated with the fifth backup session (i.e., the backup session that was created and cannot be processed at this time) is placed back into the corresponding asset queue.

[0071] When the proxy host finishes processing each of the four backup sessions, terminate the container used during the backup session processing and repeat Figure 4.2 to Figure 4.3 to process the remaining slices in the asset queue.

[0072] Separate backup sessions

[0073] Traditionally, a backup session contains a specific number of slices that are distributed to available proxy hosts based on the number of parallel streams selected by the user and the availability of the proxy hosts. Therefore, each backup session will wait for all slices in the backup session to be fully backed up before creating a new backup session. This causes parallel streams to remain unused until a new backup session begins, thereby wasting computer resources (i.e., proxy host availability). In the new method described below, each slice is scheduled as a separate backup session, allowing it to be backed up independently and in parallel with other slices (which are in their own backup sessions). In addition, in this new method, backup sessions for slices of the same asset do not need to be executed on the same proxy node as other slices associated with the asset, allowing computer resources to be maximized. In this way, proxy host resources will be fully utilized and the possibility of proxy hosts being idle will be reduced.

[0074] Figure 5.1 to Figure 5.4 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown. Figure 5.1 to Figure 5.4The method shown may be performed by, for example, a NAS data manager (202). Without departing from the present invention, Figure 1 to Figure 3.2 Other components of the system can execute Figure 5.1 to Figure 5.4 All or part of the method.

[0075] All or part of the method shown may be performed by one or more components of the system. Although the various steps in this flowchart are presented and described sequentially, it should be understood by those skilled in the art that some or all of the steps may be performed in a different order, combined or omitted, and some or all of the steps may be performed in parallel.

[0076] Figure 5.1 to Figure 5.4 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown. Figure 5.1 to Figure 5.4 The method in Figure 4.1 Occurs after the method in is executed.

[0077] Go to Figure 5.1 , in step 500, an asset queue is selected for filling. Each valid asset queue is filled with a slice ID corresponding to the slice generated in step 404 (eg, FIG. 4).

[0078] In step 502, the NAS data manager obtains a parallel configuration setting specified by, for example, a user. The parallel configuration setting specifies the maximum number of parallel streams. The user can Figure 1 , 106) configuration and / or utilization to set the number of parallel streams.

[0079] In step 504, a set of backup sessions are created based on the selected populated asset queue in step 500 and the parallel configuration settings selected by the user in step 502. In this method, each backup session created contains only one slice ID from the selected populated asset queue.

[0080] In step 506, the proxy infrastructure constraints are determined. The proxy infrastructure constraints specify the maximum number of parallel streams (ie, concurrent backup sessions) that can be executed on each proxy host.

[0081] In step 508, a reservation request is sent to the proxy infrastructure. The reservation request specifies the backup sessions generated in step 504. The number of specified backup sessions in the reservation request does not exceed the proxy infrastructure constraint determined in step 506. For example, if the proxy infrastructure constraint is 24 backup sessions per proxy host, then the reservation request cannot exceed 24 backup sessions per proxy host. Thus, if there are 28 backup sessions, the reservation request may specify 24 backup sessions to be performed on one proxy host and 4 backup sessions to be performed on a second proxy host. It will be appreciated by those skilled in the art that other methods of partitioning the backup sessions across proxy hosts may be used without departing from the present technology.

[0082] In step 510, a reservation response is received from the proxy infrastructure. The reservation response specifies that the proxy infrastructure is not able to process any of the backup sessions specified in the reservation request, is able to process some or all of the backup sessions. The reservation response may specify (i) backup sessions that can be processed (i.e., successfully reserved) and / or (ii) backup sessions that cannot be processed (i.e., unsuccessfully reserved).

[0083] In step 512, a determination is made as to whether all backup sessions in the reservation request are successfully reserved (i.e., the proxy infrastructure is able to process all backup sessions specified in the reservation request). The determination in step 512 is based on the reservation response. If all backup sessions in the reservation request are not successfully reserved, the method proceeds to step 514; otherwise, the method proceeds to step 516.

[0084] In step 514, when all backup sessions in the reservation request are not successfully reserved, the slice IDs associated with the unsuccessfully reserved backup sessions are put back into the corresponding asset queue.

[0085] In step 516, processing of all successfully scheduled backup sessions is initiated on the proxy infrastructure. In one embodiment, each backup session is associated with a proxy host (e.g., Figure 3.2 , 304) on a container (e.g., Figure 3.2 , 320A). The container includes the following functionality: (i) selecting a slice ID from the backup session, (ii) reading a slice associated with the slice ID from the primary storage device, and (iii) writing the slice to an appropriate location in the backup storage system.

[0086] Once a slice ID in a given backup session has been processed (i.e., the slice corresponding to the slice ID has been read from the primary storage device and written to the backup storage system), the container agent ( Figure 3.2 , 322A) to the agent manager ( Figure 3.1 , 302). Then, the proxy manager sends a notification to the NAS data manager ( Figure 2 , 202) issue a notice.

[0087] In one embodiment, once a container has finished processing a backup session, the container is torn down (or otherwise terminated) on the proxy infrastructure to free up resources for other containers to be instantiated.

[0088] Go to Figure 5.2 In step 518, Figure 5.2 How the NAS data manager processes the above notification from the proxy manager is described. Turning to step 518, in step 518, the NAS data manager receives a notification from the proxy infrastructure that a backup session on a container on the proxy host has been fully processed.

[0089] In step 520, a determination is made as to whether there are any remaining filled asset queues. If there are remaining filled asset queues, the process proceeds to step 522 (in Figure 5.3 , discussed below); otherwise, the process ends.

[0090] Go to Figure 5.3 , in step 522, a populated asset queue is selected.

[0091] In step 524, a reservation request is sent to the proxy infrastructure. The reservation request specifies the backup sessions generated in step 504. The number of backup sessions specified in the reservation request does not exceed the proxy infrastructure constraints determined in step 506.

[0092] In step 526, a reservation response is received from the proxy infrastructure. The reservation response specifies that the proxy infrastructure is not able to process any of the backup sessions specified in the reservation request, is able to process some or all of the backup sessions. The reservation response may specify (i) backup sessions that can be processed (i.e., successfully reserved) and / or (ii) backup sessions that cannot be processed (i.e., unsuccessfully reserved).

[0093] In step 528, a determination is made as to whether all backup sessions in the reservation request are successfully reserved (i.e., the proxy infrastructure is able to process all backup sessions specified in the reservation request). The determination in step 528 is based on the reservation response. If all backup sessions in the reservation request are not successfully reserved, the method proceeds to step 532; otherwise, the method proceeds to step 530.

[0094] In step 530, when all backup sessions in the reservation request are not successfully reserved, the slice IDs associated with the unsuccessfully reserved backup sessions are put back into the corresponding asset queue.

[0095] In step 532, processing of all successfully scheduled backup sessions is initiated on the proxy infrastructure. In one embodiment, each backup session is associated with a proxy host (e.g., Figure 3.2 , 304) on a container (e.g., Figure 3.2 , 320A). The container includes the following functionality: (i) select a slice ID from the backup session, (ii) read the slice associated with the slice ID from the primary storage device, and (iii) write the slice to the appropriate location in the backup storage system. Once step 532 is completed, the process proceeds to step 518 (at Figure 5.2 middle).

[0096] In some scenarios, new resources may become available in the proxy infrastructure, which may allow additional backup sessions to be processed. Figure 5.4 An implementation scheme for detecting and utilizing new resources in a proxy infrastructure is shown.

[0097] Go to Figure 5.4 In step 534, new proxy infrastructure capacity is detected. The new proxy infrastructure may be additional proxy hosts and / or additional resources on existing and / or new proxy hosts. The data manager may monitor the proxy infrastructure and detect new proxy infrastructure capacity based on the monitoring. Additionally or alternatively, a user (e.g., an administrator) may trigger a new proxy infrastructure capacity after they have increased the proxy infrastructure capacity. Figure 5.4 The method shown.

[0098] In step 536, a determination is made as to whether there are any remaining unprocessed backup sessions in the corresponding populated asset queue. If unprocessed backup sessions remain, the method proceeds to step 538; otherwise, the process ends.

[0099] In step 538, a determination is made as to whether the proxy infrastructure is below the parallel configuration setting operation. If the backup session is below the parallel configuration setting operation, the method proceeds to step 522 (at Figure 5.3 otherwise, the process ends.

[0100] Each time a backup session is fully processed, the Data Manager can execute Figure 5.1 to Figure 5.3 Thus, at any given time, the proxy infrastructure may be processing backup sessions associated with different assets. Figure 4.1 The method in Figure 5.1 to Figure 5.4 The methods shown are performed simultaneously.

[0101] Non-limiting examples intended to highlight various aspects of the present technology are described below.

[0102] Turning to the example, consider the following scenario: there are ten assets, and each of these assets is divided into 80 slices, for a total of 800 slices for all ten assets. Once the slices are created, the slice IDs are placed in the data manager (e.g. Figure 2 , 200). Therefore, each of the per-asset queues includes 80 slice IDs corresponding to the slices of the assets associated with the per-asset queue.

[0103] At a later point in time, the NAS data manager (e.g., Figure 2 , 202) determines that the number of parallel streams for a given asset is 28 (i.e., the parallel configuration setting), and each proxy host processes at most 24 backup sessions in parallel (i.e., the proxy infrastructure constraint). In this example, it is assumed that the proxy infrastructure includes two proxy hosts that can be used to process the slices. Therefore, the NAS data manager then selects the asset and then obtains 28 slice IDs from the corresponding asset queue to generate 28 backup sessions, each with one slice.

[0104] The NAS data manager then sends a reservation request to the proxy infrastructure, where the reservation request specifies 28 backup sessions. After receiving the reservation request, the proxy infrastructure attempts to connect to the proxy host (e.g., Figure 3.1 , 304A, 304N) create 28 containers (one container for each backup session) (for example, Figure 3.2 , 302A, 320N). In response, the proxy infrastructure creates 24 containers on one proxy host and 4 containers on a second proxy host.

[0105] When the proxy host finishes processing each of the four backup sessions, terminate the container used to process the backup session and repeat Figure 4.2 to Figure 4.3 to process the remaining slices in the asset queue.

[0106] Maximize resource utilization

[0107] Traditionally, a backup session contains a specific number of slices that are distributed to the available proxy hosts based on the number of parallel streams selected by the user and the availability of the proxy hosts. Therefore, each backup session will wait for all slices in the session to be fully backed up before creating a new backup session. This causes parallel streams to remain unused until a new backup session begins, thereby wasting computer resources. In the new method described below, slices are continuously fed from the asset queue to each proxy host once the backup of the previous slice has been completed. This method allows for maximum utilization of computer resources because individual slices are not retained in the asset queue while waiting for the previous batch of slices to be fully completed. In this way, the proxy host resources will be fully utilized and the likelihood of the proxy host being idle will be reduced.

[0108] Figure 6.1 to Figure 6.2 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown. Figure 6.1 to Figure 6.2 The method shown may be performed by, for example, a NAS data manager (202). Without departing from the present invention, Figure 1 to Figure 3.2 Other components of the system can execute Figure 6.1 to Figure 6.2 All or part of the method.

[0109] All or part of the method shown may be performed by one or more components of the system. Although the various steps in this flowchart are presented and described sequentially, it should be understood by those skilled in the art that some or all of the steps may be performed in a different order, combined or omitted, and some or all of the steps may be performed in parallel.

[0110] Figure 6.1 to Figure 6.2 A flow chart of a method for orchestrating data flow according to one or more embodiments of the present technology is shown. Figure 6.1 to Figure 6.2 The method in Figure 4.1 Occurs after the method in is executed.

[0111] Go to Figure 6.1 , in step 600, a populated asset queue is selected.

[0112] In step 602, slices (or more specifically, slice IDs) from the selected populated asset queue are assigned to one or more backup sessions, where each backup session may include up to a specified number of slice IDs and where the number of backup sessions does not exceed resource availability (e.g., the number of available proxy hosts).

[0113] In step 604, all backup sessions generated based on the populated asset queues are started on the proxy infrastructure in available containers. In one embodiment, the proxy manager determines: (i) which proxy hosts have sufficient resources to instantiate containers, which are then used to execute backup sessions and / or (ii) which proxy hosts on which containers are currently instantiated should be allocated additional backup sessions for processing. Determining whether a given proxy host has sufficient resources may include, for example, monitoring proxy hosts (also referred to as compute nodes) to determine whether a given proxy host is overutilized (i.e., operating above a threshold) or underutilized (i.e., operating below a threshold). The threshold may be related to memory and / or processor usage. If a compute node is operating above a threshold, a backup session may not be sent to the compute node for processing (or there may be a delay (e.g., a waiting period) before a backup session is sent to the compute node). If a compute node is operating below a threshold, a backup session may be preferentially sent to such compute nodes. In one embodiment, each backup session is associated with a proxy host (e.g., Figure 3.2 , 304) on a container (e.g., Figure 3.2 , 320A). The container includes the following functionality: (i) selecting a slice ID from the backup session, (ii) reading a slice associated with the slice ID from the primary storage device, and (iii) writing the slice to an appropriate location in the backup storage system.

[0114] Once a slice ID in a given backup session has been processed (i.e., the slice corresponding to the slice ID has been read from the primary storage device and written to the backup storage system), the container agent ( Figure 3.2 , 322A) to the agent manager ( Figure 3.1 , 302). Then, the proxy manager sends a notification to the NAS data manager ( Figure 2 , 202) issues a notification that the slice has been processed.

[0115] Go to Figure 6.2 In step 608, the NAS data manager receives notification from the proxy infrastructure that the backup session for the slice of the asset has completed. Therefore, the container responsible for backing up the slice of the asset is now available for the next slice in the asset queue to be filled. Figure 4.2 to Figure 4.3 and Figure 5.1 to Figure 5.4 In the method shown, the notification in step 608 indicates that a specific slice has been processed, rather than all slices in the backup session. In addition, once step 604 has been completed, the container for processing the slice will now be terminated.

[0116] In step 610 , a determination is made as to whether there are any remaining slices in the corresponding populated asset queue. If there are remaining slices in any populated asset queue, the method proceeds to step 612 ; if there are no remaining slices in the queue, the method proceeds to step 614 .

[0117] In step 612 , unprocessed slices are selected and reassigned to newly available containers. The method returns to step 604 .

[0118] In step 614, once there are no slices remaining in any asset queue, the corresponding container (ie, the container that issued the notification received in step 606) is unloaded and removed from the proxy infrastructure, and the process ends.

[0119] As discussed above, the embodiments described herein may be implemented using a computing device. Figure 7 A diagram of a computing device according to one or more disclosed embodiments is shown. The computing device (700) may include one or more computer processors (702), non-persistent storage devices (704) (e.g., volatile memory, such as random access memory (RAM), cache memory), persistent storage devices (706) (e.g., hard disk, optical drive such as compact disk (CD) drive or digital versatile disk (DVD) drive, flash memory, etc.), communication interface (712) (e.g., Bluetooth interface, infrared interface, network interface, optical interface, etc.), input device (710), output device (908), and numerous other elements (not shown) and functionality. Each of these components is described below.

[0120] In one or more embodiments, the computer processor (702) may be an integrated circuit for processing instructions. For example, the computer processor may be a single core or multi-core or micro-core processor. The computing device (700) may also include one or more input devices (710), such as a touch screen, a keyboard, a mouse, a microphone, a touch pad, an electronic pen, or any other type of input device. In addition, the communication interface (712) may include an integrated circuit for connecting the computing device (700) to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device, such as another computing device.

[0121] In one or more embodiments, the computing device (700) may include one or more output devices (708), such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input devices. The input devices and output devices may be connected locally or remotely to the computer processor (702), the non-persistent storage device (704), and the persistent storage device (906). There are many different types of computing devices, and the input devices and output devices described above may take other forms.

[0122] The problems discussed above should be understood as examples of problems solved by the embodiments described herein, and the scope of the embodiments should not be limited to solving the same / similar problems. The disclosed technology is broadly applicable to solving a range of problems beyond those discussed herein.

[0123] One or more embodiments of the present technology can be implemented using instructions executed by one or more processors of a computing device. In addition, such instructions may correspond to computer-readable instructions stored on one or more non-transitory computer-readable media.

[0124] In the above description, numerous details are set forth as examples of the embodiments described herein. Those skilled in the art (who also benefit from this detailed description) will appreciate that one or more of the embodiments described herein may be practiced without these specific details, and that numerous changes or modifications may be feasible without departing from the scope of the embodiments described herein. Certain details known to those of ordinary skill in the art may be omitted to avoid obscuring the description.

[0125] In the description of the drawings above, any component described with respect to the drawings may be equivalent to one or more similarly named components described with respect to any other drawings in the various embodiments described herein. For the sake of brevity, the description of these components will not be repeated with respect to each drawing. Therefore, each embodiment of the components of each drawing is incorporated by reference and is assumed to be optionally present in each other drawing with one or more similarly named components. In addition, according to the various embodiments described herein, any description of the components of the drawings should be interpreted as an optional embodiment, which can be implemented in addition to, in combination with, or in place of the corresponding similarly named component descriptions in any other drawings.

[0126] Throughout this application, the elements of the drawings may be labeled A to N. As used herein, the above labeling means that the element may include any number of objects and does not require that the element include the same number of elements as any other object labeled A to N. For example, a data structure may include a first element labeled A and a second element labeled N. This labeling criteria means that the data structure may include any number of elements. The second data structure, also labeled A to N, may also include any number of elements. The number of elements of the first data structure and the number of elements of the second data structure may be the same or different.

[0127] Throughout this application, ordinals (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any nouns in this application). Unless explicitly disclosed, such as by using the terms "before," "after," "single," and other such terms, the use of ordinals does not imply or create any particular ordering of elements, nor does it limit any element to only a single element. Rather, the use of ordinals is to distinguish elements. For example, a first element is different from a second element, and a first element may encompass more than one element and be after (or before) a second element in the ordering of elements.

[0128] As used herein, the phrases operatively connected or operatively connected means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact with each other in some manner. For example, the phrase 'operatively connected' may refer to any direct connection (e.g., a direct wired connection between two devices or components) or indirect connection (e.g., a wired and / or wireless connection between any number of devices or components that connect operatively connected devices). Thus, any path through which information can propagate may be considered an operative connection.

[0129] Although the embodiments described herein are described with respect to a limited number of embodiments, those skilled in the art having benefit of this detailed description will appreciate that other embodiments may be conceived without departing from the scope of the embodiments as disclosed herein. Accordingly, the scope of the embodiments described herein shall be limited solely by the appended claims.

Claims

1. A method for performing a backup operation, the method comprising: receiving a backup request for an asset; generating a plurality of slices of the asset based on the snapshot of the asset, wherein each slice relates to a separate portion of the asset; filling a queue with the plurality of slices; receiving, from the agent manager, an amount of available capacity on the agent infrastructure; dividing the plurality of slices into a plurality of backup sessions, and wherein a number of the plurality of backup sessions is based at least on an amount of the available capacity; sending a first reservation request for at least some of the amount of available capacity to the agent manager, wherein the first reservation request specifies the number of the plurality of backup sessions; making a first determination that a first portion of the plurality of backup sessions are successfully scheduled and a second portion of the plurality of backup sessions are not successfully scheduled; sending a request to initiate the first portion of the plurality of backup sessions; sending the second portion of the plurality of backup sessions to the queue based on the first determination; sending, to the agent manager, a second reservation request for the second portion of the plurality of backup sessions based on the first determination; making a second determination that the second reservation request was successful; as well as A request is sent to initiate the second portion of the plurality of backup sessions.

2. The method of claim 1, wherein the amount of available capacity is based on the availability of proxy hosts in the proxy infrastructure.

3. The method of claim 1, wherein the size of each backup session is based on user input.

4. The method of claim 1, wherein the asset is data or metadata associated with a portion of a data hierarchy.

5. The method of claim 1, further comprising: receiving a notification of completion of the backup session of the first portion of the plurality of backup sessions; In response to the notification, making a third determination that the queue contains at least one additional slice; sending a third reservation request for at least some of the amount of available capacity to the agent manager, wherein the third reservation request is based on the third determination; making a fourth determination that the third reservation request is successful; as well as A second request is sent based on the fourth determination to initiate a backup session for the at least one additional slice.

6. The method of claim 5, further comprising: receiving a second notification of success of the backup session for the at least one additional slice; After receiving the second notification, making a fifth determination that the queue is empty; as well as The backup operation is ended based on the fifth determination.

7. A method for performing a backup operation, the method comprising: receiving a backup request for an asset; After receiving the backup request, obtaining a plurality of slices of the asset, wherein each slice relates to a separate portion of the asset; filling a queue with the plurality of slices; splitting the plurality of slices into a plurality of backup sessions based on available capacity in a proxy infrastructure, wherein each backup session includes a separate portion of the slices; sending a first reservation request to the proxy infrastructure, wherein the first reservation request specifies the plurality of backup sessions; making a first determination that a first portion of the plurality of backup sessions are successfully scheduled and a second portion of the plurality of backup sessions are not successfully scheduled; sending a request to initiate said first portion of said plurality of backup sessions on said proxy infrastructure; as well as The second portion of the plurality of backup sessions is sent to the queue based on the first determination.

8. The method of claim 7, wherein generating the plurality of slices is based on a snapshot of the asset.

9. The method of claim 7, further comprising: The available capacity is received from the broker infrastructure.

10. The method of claim 9, wherein the available capacity is based on the availability of proxy hosts in the proxy infrastructure.

11. The method of claim 7, wherein the size of each backup session is based on user input.

12. The method of claim 7, wherein a container is created on the proxy infrastructure for each backup session.

13. The method of claim 12, wherein the container is terminated when a given backup session is successfully processed.

14. A non-transitory computer readable medium comprising computer readable program code which, when executed by a computer processor, enables the computer processor to perform a method for performing a backup operation, the method comprising: receiving a backup request for an asset; splitting a plurality of slices of the asset into a plurality of backup sessions based on available capacity in a proxy infrastructure, wherein each backup session includes a separate portion of the slice, wherein the plurality of slices are associated with a queue; sending a first reservation request to the proxy infrastructure, wherein the first reservation request specifies the plurality of backup sessions; making a first determination that a first portion of the plurality of backup sessions are successfully scheduled and a second portion of the plurality of backup sessions are not successfully scheduled; sending a request to initiate said first portion of said plurality of backup sessions on said proxy infrastructure; as well as The second portion of the plurality of backup sessions is sent to the queue based on the first determination.

15. The non-transitory computer-readable medium of claim 14, wherein generating the plurality of slices is based on a snapshot of the asset.

16. The non-transitory computer readable medium of claim 14, further comprising: The available capacity is received from the broker infrastructure.

17. The non-transitory computer-readable medium of claim 16, wherein the available capacity is based on availability of proxy hosts in the proxy infrastructure.

18. The non-transitory computer readable medium of claim 14, wherein the size of each backup session is based on user input.

19. The non-transitory computer-readable medium of claim 14, wherein a container is created on the proxy infrastructure for each backup session.

20. The non-transitory computer-readable medium of claim 19, wherein the container is terminated when a given backup session is successfully processed.

Images