Automatic expansion, resilience and load balancing framework for workload deployment in cloud and high performance computing environments

Abstract

The invention relates to an automatic expansion, resilience, and load balancing framework for workloads in cloud and high performance computing environments. In some embodiments, a method includes: receiving, at a framework daemon, an indication to begin executing a workload partitioned into execution units; in response to the indication, accessing, by the framework daemon, an execution context including an execution unit; associating, by the framework daemon, an identifier of the node with one of the execution units to declare execution of the execution unit by the node; executing an execution unit by the node; accessing, by the framework daemon, the execution context to determine whether the execution context includes an undeclared execution unit after the node completes execution of the execution unit; when the execution context includes an undeclared execution unit, associating, by the framework daemon process, an identifier of the node with the undeclared execution unit to declarate execution of the undeclared execution unit by the node; and executing, by the node, the execution unit.

Description

[0001] Government Support Statement

[0002] This invention was made with Government support under Contract No. H98230-15-D-0022 / 0003 awarded by the Maryland Procurement Office. The Government has certain rights in this invention. Background Art

[0003] In a computing environment, a workload is typically executed using more than one node. In such cases, a workload manager can allocate multiple nodes to execute the workload and assign portions of the workload to each of the allocated nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Some examples discussed herein will be described with reference to the drawings listed below. However, the drawings merely illustrate certain aspects or implementations of the examples described herein by way of example and are not intended to limit the scope of the claims.

[0005] Figure 1 illustrates a block diagram of an example system for implementing a workload execution framework according to one or more examples disclosed herein;

[0006] Figure 2 illustrates a block diagram of an example execution context 116 according to one or more examples disclosed herein;

[0007] Figure 3 illustrates an overview of an example method for executing a workload in an HPC environment using a workload execution framework according to one or more examples disclosed herein;

[0008] Figure 4 illustrates an overview of an example method for executing a workload in an HPC environment using a workload execution framework in the presence of a node failure according to one or more examples disclosed herein; and

[0009] Figure 5 illustrates a block diagram of a computing device according to one or more examples disclosed herein;

[0010] Figure 6 A block diagram of a computing device according to one or more examples disclosed herein is illustrated. DETAILED DESCRIPTION

[0011] High Performance Computing (HPC) workloads, workloads executed in cloud computing environments, and many other types of workloads typically include any number of workload portions (e.g., predetermined portions of an overall workload) that can be executed on any number of nodes (e.g., computing devices) deployed in an HPC environment (e.g., a data center, a cloud environment, or any other type of HPC environment). As an example, an HPC environment may include a large number of heterogeneous nodes (e.g., thousands or tens of thousands), which may have different capabilities with respect to various resources (e.g., computing resources, network resources, accelerator resources (e.g., graphics processing units (GPUs)), storage resources, or any other type of accelerator). In such an HPC environment, executing a workload may require a scheduler, workload manager, or the like that allocates resources (e.g., nodes) and schedules the various workload portions across all or any portion of the nodes in the HPC environment. A workload may be any set of operations, actions, processes, or any other activities performed by nodes included in the HPC environment within the environment.

[0012] However, certain problems may arise in an HPC environment that may prevent and / or reduce the efficient execution of a workload. One such problem is that portions of a workload to be executed are typically statically pre-assigned to nodes in the HPC environment. Such static assignments typically fail to account for differences in node performance capabilities and / or the rates at which portions of the workload can be completed in the HPC environment. Such static assignments of workload portions may result in a portion of a node that has completed its statically assigned portion of the workload being required to wait for other nodes to complete their statically assigned portion of the workload before continuing and / or completing execution of the workload.

[0013] Another problem arises in the context of node failures. Node failures are typically addressed using some form of checkpoint and restart mechanism, where the progress of the workload must be periodically checked (incurring both time and resource overhead), and when one of the set of nodes executing the workload fails, the workload must be restarted from the last checkpoint.

[0014] Yet another issue that may arise in the context of workload execution by nodes in an HPC environment is that the workload may be divided into a number of workload portions, and execution of the workload may be delayed while the necessary number of nodes sufficient to execute the workload portions become available in the HPC environment.

[0015] To at least address the above issues, examples disclosed herein include technology for providing a workload execution framework for workload execution in an HPC environment that is capable of automatically scaling, providing elasticity, and providing load balancing. In one or more examples, the workload execution framework includes: a framework monitor, which may be included in a workload manager of the HPC environment, for example; a framework daemon that executes on each node of the HPC environment for executing a workload; and a framework execution context that may be included in a shared memory / storage device accessible by each node and that stores information related to execution units (e.g., workload portions) into which the workload has been partitioned before execution.

[0016] In one or more examples, the workload can be divided into any number of execution units before executing the workload in the HPC environment. In one or more examples, an execution unit is any portion of the workload that can be executed by a node. As an example, for a workload that includes performing sparse matrix multiplication, an execution unit can include one of the matrix rows to be multiplied. In one or more examples, an execution unit is uniquely identified in a shared memory within the group of execution units. As an example, each execution unit can be numbered, have a unique identifier, etc.

[0017] In one or more examples, the execution context that can be stored can be any form of computer storage device that can be accessed by the nodes of the HPC environment that will execute the workload by executing the execution unit, for example, a networked storage (Fabric Attached Storage) device that each node is configured to access, a shared memory pool that each node is configured to access, a network attached storage device that each node is configured to access, etc. Without departing from the scope of the examples disclosed herein, the execution context can be stored in any other form of computer storage device. The execution context may include a list of execution units and an identifier of the execution unit. Without departing from the scope of the examples disclosed herein, the execution context may include any other information related to the execution unit. As an example, the execution context may include a table that lists a set of numbered execution units, and the table includes a field for the identifier of the node that executes the execution unit (discussed further below).

[0018] In one or more examples, when a workload is to be executed in an HPC environment, the workload manager can be configured to assign at least some portion of the nodes of the HPC environment for executing the workload. However, in one or more examples, the execution units partitioned by the workload are not statically assigned to the nodes. Instead, in one or more examples, the nodes assigned for workload execution are configured to claim the execution units, execute the claimed execution units, and then claim any remaining unexecuted execution units, and any node can claim any execution unit. In one or more examples, such dynamic claiming of execution units provides flexibility for the nodes, so that the workload can continue to be executed even in the case of node failure (discussed further below), and the nodes can be expanded as needed (e.g., increasing the number of nodes, reducing the number of nodes, replacing failed nodes). The number of nodes assigned to execute the workload can depend on any number of factors, such as the number of nodes available for executing the workload at a given time, the number of execution units pre-partitioned by the workload, service level agreement parameters (such as execution time), or any other relevant factors. Other factors may influence the number of nodes assigned by the workload manager to execute a workload without departing from the scope of the examples disclosed herein.

[0019] The number of nodes assigned to execute the workload can be equal to the number of execution units, or can be less than the number of execution units. As an example, the workload can be divided into thirty execution units, and based on the expected workload execution time, the workload manager can assign ten nodes to execute the workload. In one or more examples, the workload manager can assign fewer nodes to the workload to start the workload based on the current lack of node availability, and, as the workload is executed and more nodes become available, the number of nodes assigned to the workload can be increased (e.g., in order to not block the start of workload execution due to waiting for more nodes to become available while automatically scaling to meet workload execution needs).

[0020] In one or more examples, after the predetermined execution units of the workload are loaded into the execution context and a set of nodes (e.g., one or more nodes) are assigned by the workload manager to execute the workload, the nodes assigned to execute the workload can receive an indication (e.g., be signaled) to begin executing the workload.

[0021] In one or more examples, in response to such an indication, the node can execute a framework daemon that accesses an execution context (e.g., using an application programming interface (API)). The framework daemon of each node can access the execution context, select one or more execution units to be executed by the corresponding node on which the framework daemon is executing, and provide an identifier of the node associated with the one or more execution units selected for execution by the node.

[0022] As an example, for a given node, the framework daemon can select an execution unit to be executed by the node and populate a field associated with the execution unit in the execution context with the node's identifier. The node's identifier can be any form of information that uniquely identifies the node among a set of nodes executing the workload (e.g., a node number, a sequence number, or any other identifier that is unique to the node). In one or more examples, in the execution context, associating the node's identifier with one or more execution units provides an indication to each of the other nodes that the node is executing one or more execution units that are associated with the node's identifier by the framework daemon.

[0023] In one or more examples, in the absence of other factors (e.g., node failures, expansion requests discussed below), the execution of the workload will be performed by each of the nodes assigned to execute the workload, which declare at least one execution unit in the execution context and execute the execution unit. In one or more examples, once the node has completed the execution of the execution unit it has declared, the framework daemon of the node accesses the execution context again to determine whether there are any other execution units that have not yet been declared for execution. In one or more examples, if there are one or more undeclared execution units, then as described above, by associating the node's identifier with the one or more execution units, the framework daemon of the node can declare that the node is executing one or more of the undeclared execution units, thereby alerting other nodes that the node is executing the one or more execution units.

[0024] In one or more examples, if the framework daemon of the node determines that there are no remaining unclaimed execution units in the execution context, the node can enter the barrier state. In one or more examples, once all nodes have entered the barrier state, execution of the workload assigned to the node can be considered complete.

[0025] In one or more examples, since nodes that can execute execution units faster can execute additional execution units of the workload, having nodes declare execution units when they complete execution of previous workload units provides dynamic load balancing for workload execution. As an example, a group of nodes can be heterogeneous, with some nodes having more resources (e.g., GPUs, memory resources). In this way, some nodes can execute execution units of the workload faster than other nodes assigned to execute the workload. In such cases, nodes with more resources that have completed execution units can dynamically declare to execute additional execution units, rather than completing statically assigned portions of the workload and then waiting for other nodes to complete their corresponding statically assigned portions of the workload. As a result, nodes with more resources execute a larger portion of the workload than nodes with fewer resources, thereby providing implicit load balancing between nodes.

[0026] In one or more examples, one or more nodes in a set of nodes assigned by a workload manager to execute a workload may fail. In one or more examples, a framework monitor of a workload execution framework may monitor the nodes assigned to execute the workload. For example, the framework monitor may be executed as part of the workload manager, may be a separate computing device operably connected to the workload manager, and / or may be part of a separate computing device operably connected to the framework manager. In addition to other possible monitoring tasks (discussed below), the framework monitor may monitor the nodes of the framework to determine whether any node has failed.

[0027] As an example, a framework daemon executing on each node can be configured to provide periodic heartbeat signals to a framework monitor while a workload is executing. In one or more examples, if a heartbeat signal is not received from a node at an appropriate time, the framework monitor can determine that the node has failed. In one or more examples, in the event of a node failure, the framework monitor can be configured to provide a signal to a framework daemon executing on each of the other nodes that indicates that the failed node has failed.

[0028] In one or more examples, the various framework daemons can respond to instructions from the framework monitor by accessing the execution context and attempting to remove the identifier of the failed node from any execution units associated with the failed node and therefore being executed. In one or more examples, although the framework daemons from each node can attempt such actions, only one framework daemon will succeed because after the first framework daemon succeeds, the remaining framework daemons will not find the identifier of the failed node associated with any execution unit in the execution context. In one or more examples, removing the identifier associating the failed node with the execution unit has the following effect: returning the execution unit to a state that can be executed by any of the remaining non-faulty nodes (e.g., as an undeclared execution unit). Therefore, once one of the nodes completes the execution unit currently being executed by that node, that node can access the execution context and treat the execution unit previously executed by the now failed node as an undeclared execution unit and thereby declare the execution for execution, as discussed above. Therefore, even in the event of a node failure, the execution of the workload can continue, thereby providing resilience to workload execution.

[0029] In one or more examples, in the event of a node failure, the framework daemon of the node that successfully removed the association between the execution unit and the failed node in the execution context can be further configured to request resources from the workload manager that assigned nodes to execute the workload to replace the failed node (e.g., another node). As described above, if no additional nodes are available, the workload can continue to be executed by continuing to declare and execute the execution units from the execution context by the remaining non-faulty nodes. However, if resources (e.g., another node) are available to replace the failed node, the workload manager can respond to the request from the framework daemon by adding another node to the set of nodes executing the workload to replace the failed node, thereby providing additional resilience and fault recovery measures for the workload execution. In one or more examples, the new node added to the set of nodes is configured with a framework daemon (like other nodes) and continues to access the execution context and begins the above-mentioned process of declaring the execution of the execution unit.

[0030] In one or more examples, a framework monitor can be configured to monitor the execution of a workload by a set of nodes assigned by a workload manager to execute the workload. The framework monitor can monitor any aspect of the workload's execution, including, but not limited to, the length of time an execution unit waits before executing, resource contention between the set of nodes executing the workload, resource availability in the HPC environment, estimated execution completion time, and / or any other aspect of the workload's execution by the nodes. Based on such monitoring, the framework monitor can issue an extension request to the workload manager that assigns resources to the workload in the HPC environment.

[0031] As an example, a framework monitor may monitor the execution of a workload and determine that the execution rate of execution units of the workload may cause the execution time of the workload to exceed an expected or planned execution time, and, in response, the framework monitor may request from the workload manager to add additional nodes to the set of nodes executing the workload to increase the execution speed of the workload, which may be referred to as an increase request.

[0032] As another example, the framework monitor may determine that one or more nodes in a set of nodes assigned to execute a workload no longer have execution units to execute and are therefore idle, and, in response, send an indication to the workload manager that the one or more nodes can be released from the state of being assigned the workload so that they can be used to execute other workloads, which may be referred to as a removal request.

[0033] Thus, in one or more examples, monitoring of the execution of a workload on a set of nodes assigned to the workload by a framework monitor can enable automatic scaling of resource usage in an HPC environment by adding and / or removing nodes on demand to meet workload execution goals while avoiding keeping unused nodes idle.

[0034] Certain examples of the present disclosure may address certain issues that arise in the context of workload execution in an HPC environment due to static assignment of portions of the workload to resources of the HPC environment—e.g., lack of dynamic load balancing between nodes after execution begins, lack of ability of the workload manager to handle resource failure scenarios, lack of ability to scale resources to meet workload execution demands—without leaving assigned resources idle.

[0035] These and other problems are addressed by the examples disclosed herein, which provide a workload execution framework that allows a node assigned to execute a workload to claim execution units partitioned from the workload for execution by the node, claim additional execution units after completing execution of previous execution units, disassociate execution units from a failed node so that the execution units can be re-executed by other nodes, request node replacement from a workload manager in the event of a node failure, and automatically scale the resources of an HPC environment to meet workload execution needs by monitoring workload execution through a framework monitor.

[0036] Figure 1 A block diagram of an example system for implementing a workload execution framework according to one or more examples of the present disclosure is illustrated. The system may include a high performance computing (HPC) environment 100. The HPC environment 100 may include a workload manager 102 and any number of nodes 104 (in Figure 1 106, Node B 108, and Node N 110). The HPC environment may also include a workload execution framework 112. The workload execution framework 112 may include an execution context 116, a framework monitor 114 (which may be included as part of the workload manager 102), and any number of framework daemons (e.g., framework daemon A 118, framework daemon B 120, framework daemon N 122) executing on the respective nodes (e.g., Node A 106, Node B 108, Node N 110). Each of these components is described below.

[0037] In one or more examples, HPC environment 100 is any collection of resources for executing a workload. Thus, HPC environment 100 can include any amount of computing resources (e.g., any number of nodes, such as 106, 108, 110, collectively referred to as nodes 104), any amount of network resources (not shown), and any amount of storage resources (not shown), as well as any amount of other resources, devices, components, etc. (e.g., workload manager 102) that facilitate workload execution.

[0038] HPC environment 100 can be deployed in a cloud computing environment (e.g., a public cloud, a private cloud, a hybrid cloud), a data center environment, or any other computing environment that includes the resources described above. The resources of HPC environment 100 can be located in the same physical location or distributed across any number of separate physical locations, which may or may not change over time.

[0039] In one or more examples, the HPC environment 100 includes any number of nodes 104 (e.g., node A 106, node B 108, node N 110). In one or more examples, the nodes (e.g., node A 106, node B 108, node N 110) are computing devices. In one or more examples, as used herein, a computing device (e.g., any of nodes 106, 108, and 110) can be any single computing device, a group of computing devices, a portion of one or more computing devices, or any other physical, virtual, and / or logical grouping of computing resources. An example of a computing device is Figure 5 and described below.

[0040] Examples of computing devices include, but are not limited to, servers (e.g., blade servers in blade server chassis, rack servers in racks, desktop servers, any other type of server device), desktop computers, mobile devices (e.g., laptop computers, smart phones, personal digital assistants, tablet computers, automotive computing systems, and / or any other mobile computing devices), storage devices (e.g., disk drive arrays, Fibre Channel storage devices, Internet Small Computer System Interface (iSCSI) storage devices, tape storage devices, flash storage arrays, network attached storage devices, any other type of storage devices), network devices (e.g., switches, routers, multilayer switches, any other type of network device), virtual machines, virtualized computing environments, logical containers (e.g., for one or more applications), groups of containers, Internet of Things (IoT) devices, node arrays of computing resources, supercomputing devices, data centers or any portion thereof, and / or any other type of computing device having the foregoing requirements.

[0041] In one or more examples, any or all of the foregoing examples may be combined to create a system of such devices, or may be separated into separate logical devices, which may be collectively referred to as computing devices. Other types of computing devices (e.g., Figure 5 106, Node B 108, Node N 110). The HPC environment 100 may include any number and / or type of such nodes (e.g., Node A 106, Node B 108, Node N 110) in any arrangement and / or configuration without departing from the scope of the examples disclosed herein.

[0042] In one or more examples, the storage and / or memory of a computing device or computing device system (e.g., node A 106, node B 108, node N 110) can be and / or include one or more data warehouses for storing any number of data structures (which store any amount of data (e.g., information)). In one or more examples, a data warehouse is any type of storage unit and / or device (e.g., a file system, a database, a set of tables, RAM, a hard drive, a solid-state drive, and / or any other storage mechanism or medium) for storing data. Further, a data warehouse can include multiple different storage units and / or devices. Multiple different storage units and / or devices may or may not be of the same type or may or may not be located in the same physical location.

[0043] In one or more examples, any storage and / or memory of a computing device or computing device system and / or network device may be considered, in whole or in part, to be a non-transitory computer-readable medium storing software and / or firmware that, when executed by one or more processors, causes the one or more processors to perform operations according to one or more examples disclosed herein.

[0044] As used herein, HPC environment 100 may include any number of nodes 104, which may be considered, individually or collectively, as computing devices. All or any portion of the computing devices may be the same type or different types of computing devices.

[0045] In one or more examples, the HPC environment 100 includes a workload manager 102. In one or more examples, the workload manager 102 is all or any portion of a computing device (as described above). In one or more examples, the workload manager 102 is configured to, at least in part, manage the deployment and execution of workloads in the HPC environment 100. Thus, in one or more examples, the workload manager 102 is configured to allocate any number of nodes (e.g., Node A 106, Node B 108, Node N 110) for executing a given workload, as well as to configure, set up, provision, or otherwise make available any other portions of the HPC environment 100 that may be required to execute the workload (e.g., network resources, storage resources, or any other type of resource).

[0046] In one or more examples, the workload manager is operably connected to each of the nodes 104 (e.g., node A 106, node B 108, node N 110) of the HPC environment 100. In one or more examples, when a workload is to be executed in the HPC environment 100, the workload manager 102 can be configured to assign multiple nodes (e.g., nodes 104) to execute the workload. The number of nodes assigned to execute the workload can depend on any number of factors, such as the number of nodes available at a given time to execute the workload, the number of execution units into which the workload is pre-partitioned, service level agreement parameters (e.g., execution time), and / or any other relevant factors. Other factors may affect the number of nodes assigned by the workload manager 102 to execute the workload without departing from the scope of the examples disclosed herein.

[0047] In one or more examples, a workload execution framework 112 is deployed within the HPC environment 100. In one or more examples, the workload execution framework 112 is a collection of components that include a framework monitor 114, any number of framework daemons (e.g., 118, 120, 122), and an execution context 116, each of which is described below. Figure 1 As shown, the workload execution framework 112 is represented by a dashed box, which encompasses the aforementioned set of components, including the framework monitor 114, any number of framework daemons (e.g., 118, 120, 122), and the execution context 116. While the workload execution framework 112 encompasses the aforementioned components, it may not fully encompass devices that include such components. As an example, the workload execution framework 112 may include the framework monitor 114, but not the portion of the workload manager 102 that includes the framework monitor 114, which is outside of the framework monitor 114. As another example, the workload execution framework 112 may include the framework daemons 118, 120, and 122, but not the portion of their respective nodes 106, 108, and 110 other than the framework daemons. In one or more examples, the workload execution framework 112 facilitates the execution of workloads according to the examples disclosed herein. Thus, the various components of the workload execution framework 112 operate in conjunction with each other to provide resilient (e.g., capable of handling node failures), load-balanced, and automatically scalable workload execution.

[0048] In one or more examples, before executing the workload in the HPC environment 100 using the workload execution framework 112, the workload can be divided into any number of execution units. In one or more examples, an execution unit is any part of a workload that can be executed by a node (e.g., node A 106, node B 108, node N 110) of the HPC environment 100. As an example, for a workload that includes multiple operations that are executed as independent processes, an execution unit can include one of these processes. In one or more examples, an execution unit is uniquely identified within a group of execution units. As an example, each execution unit can be numbered, have a unique identifier, or can be uniquely identified within the group of execution units using any other appropriate technology for distinguishing an item from other items in a group of items. In one or more examples, the execution units divided by the workload are not statically assigned to the nodes before the workload is executed.

[0049] In one or more examples, the workload execution framework 112 includes an execution context 116. In one or more examples, the execution context 116 is any component configured to store information and at least operatively connected to a node 104 assigned by the workload manager 102 to execute the workload. In one or more examples, the execution context can be any form of storage accessible by the nodes of the HPC environment that will execute the workload by executing execution units, such as a fabric attached storage device that each node is configured to access, a shared memory pool that each node is configured to access, a network attached storage device that each node is configured to access, and / or any other form of persistent or non-persistent memory or storage accessible by each of the nodes 104.

[0050] In one or more examples, the execution context 116 is configured to store information related to the above-mentioned execution units that have been partitioned from the workload (but have not been pre-assigned to any node before the nodes 104 of the HPC environment 100 execute the workload). The execution context 116 may include a list of execution units and identifiers of the execution units, and may also include a list of identifiers. The execution context may include any other information related to the execution units without departing from the scope of the examples disclosed herein. As an example, the execution context may include a table that lists a set of numbered execution units and includes a field for the identifiers of the nodes for executing these execution units (discussed further below). Examples of execution context 116 are described below. Figure 2 This is discussed further in the description of .

[0051] In one or more examples, the workload execution framework 112 includes any number of framework daemons (e.g., 118, 120, 122). In one or more examples, the framework daemon is any hardware, software, and / or firmware configured on, as part of, and / or executing on one of the nodes 104, where the node 104 is assigned by the workload manager 102 to execute a workload. In one or more examples, the framework daemons (e.g., 118, 120, 122) are configured to use one or more APIs to interact with other components in the HPC environment 100 and the workload execution framework 112 (e.g., the execution context 116, the framework monitor 114, and / or the workload manager 102).

[0052] In one or more examples, the framework daemon (e.g., 118, 120, 122) is configured to receive an indication that workload execution is to begin, and in response to such indication, access the execution context 116 to declare one or more execution units to be executed by the corresponding node on which the framework daemon is deployed. The operations and actions performed by the framework daemon (e.g., 118, 120, 122) are described below. Figure 3 and Figure 4 This is discussed further in the description of .

[0053] In one or more examples, the workload execution framework 112 includes a framework monitor 114. In one or more examples, the framework monitor 114 is all or any portion of a computing device (as described above). As an example, the framework monitor 114 can be part of the workload manager 102 (e.g., Figure 1 ). Alternatively, the framework monitor 114 can be separate from and operably connected to the workload manager 102. In one or more examples, the framework monitor 114 is operably connected to each of the nodes 104, and thereby to each of the framework daemons (e.g., 118, 120, 122) executing thereon.

[0054] In one or more examples, the framework monitor 114 is configured to monitor the execution of workloads on the nodes 104. Thus, in one or more examples, the framework monitor 114 can be configured to monitor the nodes 104 for any node failures. As an example, the framework daemon (e.g., 118, 120, 122) executing on each node can be configured to provide periodic heartbeat signals to the framework monitor 114 while the workload is executing. In one or more examples, if a heartbeat signal is not received from a node at an appropriate time, the framework monitor 114 can determine that the node has failed. Monitoring of node failures by the framework monitor 114 will be discussed below. Figure 4 This is discussed further in the description of .

[0055] In one or more examples, the framework monitor 114 can be configured to monitor the execution of a workload by a set of nodes 104 assigned by the workload manager 102 to execute the workload. The framework monitor 114 can monitor any aspect of the workload's execution, including, but not limited to, the length of time an execution unit waits before executing, resource contention between the set of nodes 104 executing the workload, resource availability in the HPC environment 100, estimated execution completion time, and / or any other relevant aspect of the workload's execution on the nodes. Based on such monitoring, the framework monitor 114 can issue an expansion request to the workload manager 102, which assigns resources to the workload in the HPC environment 100.

[0056] As an example, the framework monitor 114 may monitor the execution of a workload and determine that the execution rate of an execution unit of the workload may cause the execution time of the workload to exceed an expected or planned execution time, and in response, issue a scaling request to the workload manager 102 to add additional nodes to the set of nodes 104 executing the workload (which may be referred to as an increase request) to increase the execution speed of the workload.

[0057] As another example, the framework monitor 114 may determine that one or more nodes in a set of nodes 104 assigned to execute a workload no longer have execution units to execute and are therefore idle, and, in response, send an indication to the workload manager 102 that the one or more nodes can be released from the state of being assigned the workload (which may be referred to as a removal request) so that they can be used to execute other workloads.

[0058] Thus, in one or more examples, monitoring of the execution of a workload on a set of nodes 104 assigned to the workload by the framework monitor 114 can enable automatic scaling of resource usage in the HPC environment 100 by adding and / or removing nodes as needed to meet workload execution goals while avoiding leaving unused nodes idle.

[0059] Although Figure 1 A particular configuration of components is shown, but other configurations may be used without departing from the scope of the examples disclosed herein. Figure 1 Certain components are shown as part of the same device, but any component may be grouped into a collection of one or more components that can exist and perform as part of any number of independent and operatively connected devices. As another example, a single component may be configured to perform the operations performed by Figure 1 As another example, only certain components of the HPC environment 100 are shown for clarity. However, the HPC environment 100 may include any number of additional components (e.g., network devices, storage devices, management devices, etc.) without departing from the scope of the examples disclosed herein. Thus, the examples disclosed herein should not be limited to Figure 1 The component configuration shown in .

[0060] Figure 2 1 illustrates a block diagram of an example execution context 216 according to one or more examples disclosed herein. Figure 2 As shown, execution context 216 includes representations of any number of execution units (e.g., execution unit A 200, execution unit B 202, execution unit N 204) and any number of node identifiers (e.g., node identifier A 206, node identifier B 208). Each of these components is described below.

[0061] In one or more examples, the execution context 216 is Figure 1 , and as described above. Thus, an execution context 216 is any memory or storage device configured to store any type of information in any form. An execution context can be part of a computing device (as described above) or a computing device (e.g., Figure 1 Storage devices and / or memories accessible to the node 104 shown.

[0062] In one or more examples, the execution context 216 is configured to store (at least temporarily) any type of data structure capable of associating information with one another. In one or more examples, the data structures stored in the execution context 216 are configured to allow the execution units of a workload to be associated with the nodes that declared the execution units (e.g., Figure 1 In one or more examples, the node is associated with the node 104 shown. Figure 1 1 and 2. In one embodiment, the framework daemon processes (framework daemon A 118, framework daemon B 120, and framework daemon N 122) access execution context 216 to locate the execution unit represented therein and declare the execution unit for execution on the node of the framework daemon process. In one or more examples, the framework daemon process can declare the execution unit by associating the node identifier (e.g., 206, 208) of the node on which the framework daemon process is executed with the execution unit represented in execution context 216.

[0063] As an example, for a given node, the framework daemon can select an execution unit to be executed by the node and populate fields associated with the execution units (e.g., execution unit A 200, execution unit B 202, execution unit N 204) in the execution context with the node's identifier (e.g., node identifier A 206, node identifier B 208). The node's identifier (e.g., node identifier A 206, node identifier B 208) can be any form of information (e.g., node number, sequence number) that uniquely identifies a node in a set of nodes executing a workload. In one or more examples, associating the node's identifier with one or more execution units provides an indication to each of the other nodes in the execution context 216 that the node is executing one or more execution units that are associated with the node's identifier by the framework daemon.

[0064] In one or more examples, some portion of the execution units in the execution context 216 may be undeclared, such as Figure 2 Execution unit N 206 is shown. In one or more examples, in various circumstances, an execution unit may be unclaimed. As an example, when a workload is divided into more execution units than there are nodes assigned to execute the workload, after each node claims an execution unit, a portion of the execution units will remain as unclaimed execution units, and these unclaimed execution units will be claimed by the node when the node completes execution of the previously claimed execution unit. As another example, an execution unit being executed by a failed node can return to an unclaimed state (see below for Figure 4description).

[0065] In one or more examples, although Figure 2 2, but execution context 216 can be configured to represent an indication that an execution unit has been executed in the data structures of the execution unit and the associated node that executed the execution unit, and / or to remove the representation of the execution unit from execution context 216 when the execution unit is completed by the node. In one or more examples, execution of the workload can be considered complete when all execution units originally included in execution context 216 are executed by the node assigned to execute the workload.

[0066] Figure 3 An overview of an example method 300 for executing a workload in an HPC environment using a workload execution framework according to one or more examples disclosed herein is illustrated. The method 300 may be implemented at least in part by a workload execution framework (e.g., Figure 1 The workload execution framework 112 shown), or any component of the workload execution framework (e.g., Figure 1 Frame Monitor 114, Figure 1 Any framework daemon (e.g., 118, 120, 122), Figure 1 Execution context 116), and / or Figure 1 The workload manager 102 is used to execute.

[0067] Although Figure 3 The various steps in the flowchart shown are presented and described in sequence, but some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, and some or all of the steps may be combined or omitted. Figure 3 and / or Figure 4 The other steps are executed in parallel.

[0068] In step 302, method 300 includes: in a workload execution framework (e.g., Figure 1 The workload execution framework 112) of the framework daemon (e.g., Figure 1The framework daemon 118, 120, 122 of the workload manager receives an indication to start executing the workload, and the framework daemon executes on the first node of a group of nodes assigned by the workload manager to execute the workload. In one or more examples, before executing the workload, the workload is divided into multiple execution units, and the execution units are not statically assigned to the node before the corresponding framework daemon of the node receives the indication to start workload execution. In one or more examples, the indication is received from any source (e.g., a workload manager, a framework monitor, a user device and / or any other source) that can provide an indication that workload execution is about to begin. In one or more examples, such an indication is received at each framework daemon of each node assigned to execute the workload. The indication can take any form that can signal a node to start workload execution. As an example, the indication can be sent as one or more network packets over a network to the framework daemon of the node to execute the workload.

[0069] In step 304, the method 300 includes: in response to the indication, accessing the execution context (eg, Figure 1 Execution context 116, Figure 2 216) that includes representations of multiple execution units. As an example, the framework daemon can access the execution context via an operable network connection between the node on which the framework daemon is executed and the execution context. In one or more examples, as described above in Figure 1 and Figure 2 As discussed in the description of , an execution context is any storage device and / or memory that is operably connected to a set of nodes assigned to execute a workload. In one or more examples, the execution context includes a data structure that includes representations of execution units into which the workload was partitioned before execution of the workload began. In one or more examples, the data structure is configured to include a field associated with each execution unit of the workload, and an identifier of a node can be stored in the field, which can have the effect of declaring the execution of the execution unit by the node identified by the node identifier.

[0070] In step 306, method 300 includes associating, by the framework daemon, a first identifier of the first node with a first execution unit in a plurality of execution units in an execution context to indicate that the first execution unit is to be executed by the first node. In one or more examples, the framework daemon may associate the node on which the framework daemon is executing with the execution unit by populating a field in the execution context associated with the execution unit with the identifier of the node. The identifier of the node may be any information item having the property of uniquely identifying a node in a set of nodes assigned (e.g., by a workload manager) to execute the workload.

[0071] At step 308, method 300 includes executing, by a first node, a first execution unit. In one or more examples, executing the execution unit on the node includes obtaining any information required to execute the execution unit and then executing the execution unit. As an example, if the workload is to perform sparse matrix multiplication, the execution unit may include multiplying rows of the matrix. Thus, executing the execution unit may include obtaining the rows of the matrix and the elements to be multiplied by the row, thereby enabling the node to perform the multiplication.

[0072] In step 310, method 300 includes: after the first node completes execution of the execution unit, accessing the execution context by the framework daemon to determine whether the execution context includes any undeclared execution units of the plurality of execution units. In one or more examples, the execution context includes all execution units in the workload to be executed. In one or more examples, the execution unit can be declared for execution by the node by being associated with a node identifier, or the execution unit can be undeclared. If no node has yet declared the execution of the execution unit, or if the node that declared the execution of the execution unit fails before completing execution of the execution unit and the execution unit returns to an undeclared state, then the execution unit can be undeclared (described below). Figure 4 (discussed further in the description of ).

[0073] In step 312, method 300 includes determining whether the execution context includes any undeclared execution units. In one or more examples, this determination is made by a framework daemon executing on a node that has completed execution of an execution unit for the workload. In one or more examples, this determination is made by the framework daemon evaluating the execution context to determine whether any execution units included and / or represented in the execution context are undeclared (e.g., not claimed by another node). In one or more examples, if there are no remaining undeclared execution units in the execution context, the method proceeds to step 318. In one or more examples, if there are one or more undeclared execution units, the method proceeds to step 314.

[0074] In step 314, the method includes: when it is determined at step 312 that the execution context includes an undeclared second execution unit, the framework daemon associates the first identifier of the first node with the second execution unit to declare that the second execution unit is executed by the first node. In one or more examples, the framework daemon can associate the node on which the framework daemon is executed with the second execution unit by filling a field in the execution context associated with the second execution unit with the identifier of the node. The identifier of the node can be any information item having the following properties: it uniquely identifies a node in a set of nodes assigned (e.g., by a workload manager) for executing the workload. In one or more examples, allowing a node to declare execution of additional execution units after completing execution of other execution units makes it possible to implicitly load balance the execution of the workload, because a node with more resources and capable of completing execution of the execution unit faster, or a node that declares execution of an execution unit that requires less time to execute, can declare execution of the additional execution unit.

[0075] In step 316, method 300 includes: executing the second execution unit by the first node. In one or more examples, executing the execution unit on the node includes obtaining any information required to execute the execution unit and then executing the execution unit. As an example, if the workload is to execute one of a set of related processes, the execution unit may include one or more processes to be executed. Thus, executing the execution unit may include obtaining any information required to execute the process and then executing the process. In one or more examples, after step 316, the method can return to step 310, in which the framework daemon process of the node that completed the execution of the execution unit accesses the execution context to determine whether there are any undeclared execution units therein.

[0076] At step 318, the method includes entering a barrier state. In one or more examples, the barrier state is a state in which a node has no more execution units to execute and can be used, for example, as a form of synchronization. In one or more examples, the workload executed by a set of nodes assigned to execute the workload can be considered complete when each node in the set of nodes has entered the barrier state.

[0077] Figure 4 An overview of an example method 400 for executing a workload in an HPC environment using a workload execution framework in the event of a node failure according to one or more examples disclosed herein is illustrated. The method 400 may be performed at least in part by a workload execution framework (e.g., Figure 1 The workload execution framework 112 shown), or any component of the workload execution framework (e.g., Figure 1 Frame Monitor 114, Figure 1 Any framework daemon (e.g., 118, 120, 122), Figure 1 Execution context 116), and / or Figure 1 The workload manager 102 is used to execute.

[0078] Although Figure 4 The various steps in the flowchart shown are presented and described in sequence, but some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, and some or all of the steps may be combined or omitted. Figure 3 and / or Figure 4 The other steps are executed in parallel.

[0079] In step 402, the method 400 includes: a frame monitor (e.g., Figure 1 The framework monitor 114) monitors the workload on a set of nodes (e.g., Figure 1 In one or more examples, the framework monitor is configured to monitor the execution of the workload to observe various characteristics of the workload execution, including but not limited to the status of the node (e.g., faulty or non-faulty). Monitoring can be performed using an operable connection between the framework monitor and each of the corresponding nodes.

[0080] In step 404, method 400 includes determining, by the framework monitor, whether any node has failed. In one or more examples, the framework monitor may determine that a node has failed using any suitable technique for detecting node failures. As an example, a framework daemon on a node may be configured to send periodic heartbeat signals to the framework monitor, and failure to receive a heartbeat signal from a node at an appropriate time may indicate a node failure. As another example, the framework monitor may be configured to periodically send an echo message (ping) to the node, and failure to respond may indicate a node failure. As another example, the framework monitor may be configured to periodically access the node, and failure to access the node may indicate a node failure. Other techniques for confirming whether a node has failed may be used without departing from the scope of the examples disclosed herein. In one or more examples, if the framework monitor does not determine that a node has failed, the method returns to step 402, and the framework monitor continues to monitor the node during execution of the workload. In one or more examples, if the framework monitor determines that a node has failed, the method proceeds to step 406.

[0081] In step 406, method 400 includes: in response to determining that the second node has failed, the framework monitor sends an indication that the second node has failed to the corresponding framework daemon of the non-faulty node in the set of nodes. In one or more examples, the indication includes a second identifier of the second node (e.g., the faulty node). The indication can take any form that can convey to the framework daemon that the node has failed. In one or more examples, the indication includes at least an indication that the node has failed and an identifier of the faulty node. In one or more examples, the identifier of the faulty node is the same as the following: the identifier of the node that the faulty node is configured to use when declaring an execution unit in the execution context. Additionally or alternatively, in one or more examples, the identifier of the node in the indication from the framework monitor received at the framework daemon is different from the identifier used by the faulty node to declare the execution unit, but the framework daemon is configured to use the identifier received in the indication to determine the identifier used by the faulty node to declare the execution unit.

[0082] In step 408, the method includes: accessing the execution context by the corresponding framework daemon to attempt to disassociate the second node from any execution units in the execution context. In one or more examples, in response to the indication received in step 406, each framework daemon is configured to attempt to access the execution context and disassociate the faulty node from any execution units in the faulty node that are associated with the faulty node (e.g., via association with an identifier of the faulty node). As an example, the framework daemon can be configured to use an API to access the execution context and evaluate the declared execution units therein to determine whether the identifier of the faulty node is associated with any of the declared execution units.

[0083] In step 410, method 400 includes: successfully disassociating, by one of the corresponding framework daemons, the second node from any execution unit included in the execution context. In one or more examples, although each framework daemon attempts to disassociate the faulty node from any execution unit, only one framework daemon will succeed because after one framework daemon succeeds, the faulty node will no longer be associated with any execution unit. In one or more examples, successfully disassociating the faulty node from the execution unit includes: removing an identifier of the faulty node from the execution context at any location in the execution context where the identifier of the faulty node is associated with the execution unit. In one or more examples, disassociating the execution unit from the faulty node (e.g., by removing the identifier of the faulty node from its association with the execution unit) causes the execution unit to return to an undeclared state, and another node can then claim to execute the execution unit.

[0084] In step 412, method 400 includes: a framework daemon in the corresponding framework daemon that successfully disassociates the second node from any execution unit included in the execution context sends an extended replacement request to the workload manager, and when the request is sent in response to a node failure, the request can be referred to as a replacement request. In one or more examples, the replacement request is a request to replace one node in a group of nodes. As an example, the replacement request can be a request to replace a second node in the group of nodes after the second node fails. In one or more examples, the framework daemon is configured to attempt to disassociate the failed node from any execution unit in the execution context in response to the indication received in step 406, and is further configured to request a replacement node from the workload manager if it successfully becomes the framework daemon that performs the disassociation. In one or more examples, only having successful framework daemons send node replacement requests to the workload manager prevents the workload manager from receiving redundant requests. In one or more examples, although in Figure 4 Not shown, the workload manager may respond to the request by attempting to replace the node. In one or more examples, if no available nodes exist in the HPC environment, the workload manager may not replace the node. In such cases, workload execution may continue regardless because the remaining non-faulty nodes assigned to execute the workload will continue to execute the execution units and then check for unclaimed execution units upon completion of execution of the previous execution units (see above for Figure 3 In one or more examples, upon receiving the request, or later when additional nodes become available in the HPC environment, the workload manager can respond to a node replacement request by allocating an additional node to replace a failed node in the set of nodes assigned to execute the workload. Thus, the examples described herein provide resiliency for workload execution by automatically replacing failed nodes or allowing non-failed nodes to continue executing the workload even after a node failure.

[0085] Figure 5 A block diagram of a computing device according to one or more examples of the present disclosure is illustrated. As discussed above, the examples described herein can be implemented using a computing device. For example, Figure 1 All or any portion of the illustrated components (e.g., workload manager 102, framework monitor 114, framework daemons (e.g., 118, 120, 122), nodes 104, execution context 116), and execution context 216 may be implemented, at least in part, using one or more computing devices, and Figure 3 and Figure 4 All or any portion of the illustrated method may be performed using one or more computing devices, such as computing device 500 .

[0086] In one or more examples, a computing device (e.g., computing device 500) is any device, part of a device, or any collection of devices capable of electronically processing instructions, and may include, but is not limited to, any of the following: one or more processors (e.g., components comprising circuitry) (e.g., processor 502), memory (e.g., random access memory (RAM)) (not shown), (multiple) input and output devices (e.g., non-persistent storage 506), non-volatile storage hardware (e.g., solid-state drive (SSD), persistent memory (PMem) device, hard disk drive (HDD) (not shown)), one or more physical interfaces (e.g., network ports, storage ports) (e.g., persistent storage 506), any number of other hardware components (not shown), and / or any combination thereof. As used herein, a processor can be any component that can be configured to execute an operation, process, thread, etc. Examples of processors include, but are not limited to, a central processing unit (CPU), a multi-core CPU, an application-specific integrated circuit (ASIC), an accelerator (e.g., a graphics processing unit (GPU)), and a field programmable gate array (FPGA).

[0087] Computing device 500 may include a communication interface 512 (e.g., a Bluetooth interface, an infrared interface, a network interface, an optical interface, or any other type of communication interface), an input device 510, an output device 508, and many other elements (not shown) and functions. Each of these components is described below.

[0088] In one or more examples, the computer processor(s) 502 may be integrated circuits for processing instructions. For example, the computer processor(s) may be one or more cores or microcores of a processor. The processor 502 may be a general-purpose processor configured to execute program code included in software executing on the computing device 500. The processor 502 may be a special-purpose processor in which certain instructions are incorporated into the processor design. The processor 502 may be an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a data processing unit (DPU), a tensor processing unit (TPU), an associative processing unit (APU), a vision processing unit (VPU), a quantum processing unit (QPU), and / or various other processing units using specialized hardware (e.g., a field programmable gate array (FPGA), a system on a chip (SOC), a digital signal processor (DSP)). Although in Figure 5 Only one processor 502 is shown, but computing device 500 may include any number of processors without departing from the scope of the examples disclosed herein.

[0089] The computing device 500 may also include one or more input devices 510, such as a touch screen, keyboard, mouse, microphone, touchpad, electronic pen, motion sensor, or any other type of input device. The input device 510 may allow a user to interact with the computing device 500. In one or more examples, the computing device 500 may include one or more output devices 508, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touch screen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, an external storage device, or any other output device. One or more of the output devices may be the same as or different from the input device(s). The input and output device(s) may be connected locally or remotely to the computer processor(s) 502, the non-persistent storage device 504, and the persistent storage device 506. Many different types of computing devices exist, and the input and output device(s) described above may take other forms. In some instances, a multimodal system may allow a user to provide multiple types of input / output to communicate with the computing device 500.

[0090] Further, the communication interface 512 can facilitate connecting the computing device 500 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or any other type of network) and / or another device (e.g., another computing device). The communication interface 512 can use any type and / or technology of wired and / or wireless transceivers to perform or facilitate receiving and / or transmitting wired or wireless communications. Examples include, but are not limited to, utilizing an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, fiber optic ports / plugs, proprietary wired ports / plugs, Wireless signal transmission, BLE wireless signal transmission, Wireless signal transmission, RFID wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 WiFi wireless signal transmission, WLAN signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 512 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing device 500 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' GPS, Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operate on any particular hardware arrangement, and therefore as hardware or firmware arrangements evolve, the basic features herein may be readily substituted for improved hardware or firmware arrangements.

[0091] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (one or more) instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals transmitted wirelessly or via a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media such as CDs or DVDs, flash memories, memories, or memory devices. Code and / or machine-executable instructions may be stored on a computer-readable medium, and the code and / or machine-executable instructions may represent any combination of a process, function, subroutine, program, routine, subroutine, module, software package, class, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or transmitted via any suitable means (including memory sharing, message passing, token passing, network transmission, etc.).

[0092] All or any portion of the components of the computing device 500 may be implemented in circuits. For example, the components may include and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, FPGAs, CPUs, and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some aspects, computer-readable storage devices, media, and memories may include wired or wireless signals including bitstreams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0093] Figure 6 FIG2 illustrates a block diagram of a computing device 600 according to one or more examples disclosed herein. The computing device 600 is previously described in Figure 5 An example of a computing device 500 is described in the description of FIG. Figure 1 、 Figure 2 、 Figure 3 、 Figure 4 and Figure 5 As discussed in the description of FIG, the computing device 600 can be used to implement Figure 1 and Figure 2 All or any part of the various components shown, and, for example, as described above Figure 1 workload manager 102, Figure 1 The framework monitor 114 of the workload manager 102, Figure 1 Execution context 116, Figure 1 Node 106, Figure 1 The framework daemon process 118 of the node A 116, Figure 1 Node B, Figure 1 The framework daemon 120 of the node B 108, Figure 1 Node N 110, Figure 1 Framework daemon N 122 of node N 110 and / or Figure 2 Execution context 216 (or any component thereof).

[0094] The computing device 600 may include one or more processors 602 and a memory 604. The memory 604 may include a non-transitory computer-readable medium storing programs for execution by one or more of the one or more processors 702. In this embodiment, one or more modules within the computing device 600 may be partially or fully implemented as software for performing any of the functions described in this disclosure. The computing device 600 may, for example, be configured to perform operations by executing instructions included in the memory 604 and executed by the one or more processors 602. Figure 3 The method 300 shown in and described above.

[0095] For example, memory 604 may include instructions 606 for receiving, at a framework daemon of a workload execution framework, an indication to begin executing a workload, the framework daemon executing on a first node of a set of nodes assigned by a workload manager to execute the workload, wherein the workload is divided into a plurality of execution units (e.g., as described above with reference to Figure 3 (as described in step 302).

[0096] The memory 604 may also include instructions 608 to access, by the framework daemon in response to an instruction, an execution context comprising a plurality of execution units (e.g., as described above with reference to Figure 3 (as described in step 304).

[0097] The memory 604 may also include instructions 610 to declare, by the framework daemon, that the first execution unit is executed by the first node by associating the first identifier of the first node with the first execution unit of the plurality of execution units (e.g., as described above with reference to Figure 3 (as described in step 306).

[0098] The memory 604 may also include instructions 612 to execute, by the first node, a first execution unit (e.g., as described above with reference to Figure 3 308).

[0099] The memory 604 may further include instructions 614 to, after the first node completes execution of the first execution unit, determine by the framework daemon process whether the execution context includes a second execution unit that is not declared in the plurality of execution units by accessing the execution context (e.g., as described above with reference to Figure 3 310 and 312).

[0100] The memory 604 may also include instructions 616 to, upon determining that the execution context includes an unclaimed second execution unit, claim, by the framework daemon, execution of the second execution unit by the first node by associating the first identifier of the first node with the second execution unit (e.g., as described above with reference to Figure 3 (as described in step 314).

[0101] The memory 604 may also include instructions 618 to execute, by the first node, a second execution unit (e.g., as described above with reference to Figure 3 316).

[0102] In the description above, many details are set forth to illustrate the examples described herein. Those skilled in the art (who also benefit from this disclosure) will understand that one or more of the examples described herein can be practiced without these specific details, and that many variations or modifications can be made without departing from the scope of the examples described herein. Certain details known to those of ordinary skill in the art may be omitted to avoid obscuring the description.

[0103] Specific details are provided in the description above to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects and examples can be put into practice without these specific details. For clarity of explanation, in some instances, the present technology can be presented as including functional blocks, which can include devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, additional components can also be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form to avoid blurring these aspects in unnecessary details. In other instances, known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid blurring the various aspects of the examples disclosed herein.

[0104] Individual aspects may be described above as processes or methods, wherein the processes or methods are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe operations as a sequential process, many operations may be performed in parallel or simultaneously. In addition, the order of the operations may be rearranged. When the operations of a process are completed, the process may terminate, but may have additional steps not included in the figure. A process may correspond to a method, function, flow, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0105] The process and method according to the above examples can be implemented using computer-executable instructions stored in a computer-readable medium or otherwise obtained from a computer-readable medium. For example, such instructions may include instructions and data that enable a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or function group or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or function group. The partial computer resources used can be accessed through a network. For example, a computer-executable instruction can be a binary file, an intermediate format instruction (such as assembly language, firmware, source code, etc.). The example of a computer-readable medium that can be used to store instructions, information used, and / or information created during the method according to the example includes a disk or optical disk, a flash memory, a USB device equipped with a non-volatile memory, a network storage device, etc.

[0106] In the description of the drawings above, in the various examples described herein, any component described with respect to a drawing may be equivalent to one or more identically or similarly named and / or numbered components described with respect to any other drawing. For the sake of brevity, the description of these components may not be repeated for each drawing. Therefore, each and all examples of components of each drawing are incorporated by reference and are assumed to be optionally present in each other drawing with one or more identically or similarly named and / or numbered components. In addition, according to the various examples described herein, any description of components of a drawing should be interpreted as an optional example, which can be implemented as a supplement to, in combination with, or in place of the examples described with respect to the corresponding one or more identically or similarly named and / or numbered components in any other drawing.

[0107] Throughout the application, ordinal numbers (e.g., first, second, third, etc.) may be used as adjectives for elements (i.e., any noun in the application). The use of ordinal numbers does not imply or create any particular order of elements, nor does it limit any element to being only a single element, unless explicitly disclosed, such as by the use of the terms "before," "after," "single," and other such terms. Quite the contrary, ordinal numbers are used to distinguish elements. For example, a first element is different from a second element, and a first element may contain more than one element and be after (or before) a second element in the order of elements.

[0108] As used herein, the phrase "operatively connected" or "operative connection" means that there is a direct or indirect connection between elements / components / devices that allows the elements to interact with each other in some manner. For example, the phrase "operatively connected" can refer to any direct connection (e.g., a direct wired connection between two devices or components) or indirect connection (e.g., a wired and / or wireless connection between any number of devices or components that connect operatively connected devices). Thus, any path that information can take can be considered an operative connection.

[0109] Although the examples discussed herein have been described with respect to a limited number of examples, those skilled in the art, having benefit of this disclosure, will appreciate that other examples can be devised that do not depart from the scope of the examples disclosed herein. Accordingly, the scope of the examples described herein should be limited only by the appended claims.

Claims

1. A system comprising: one or more processors; as well as One or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving, at a framework daemon of a workload execution framework, an indication to begin executing a workload, the framework daemon executing on a first node of a set of nodes assigned by a workload manager to execute the workload, wherein the workload is divided into a plurality of execution units; In response to the indication, accessing, by the framework daemon process, an execution context including the plurality of execution units; Declaring, by the framework daemon process, that the first execution unit is executed by the first node by associating a first identifier of the first node with a first execution unit among the plurality of execution units; executing the first execution unit by the first node; After the first node completes execution of the first execution unit, the framework daemon process determines, by accessing the execution context, whether the execution context includes a second execution unit that is not declared in the plurality of execution units; When it is determined that the execution context includes the undeclared second execution unit, declaring, by the framework daemon process, execution of the second execution unit by the first node by associating the first identifier of the first node with the second execution unit; and The second execution unit is executed by the first node.

2. The system of claim 1, wherein: Execution of the instructions further causes the one or more processors to: determining, by a framework monitor of the workload execution framework, that a second node in the set of nodes that is executing the workload has failed; In response to determining that the second node has failed, the framework monitor sends a second indication that the second node has failed to the framework daemons corresponding to the non-failed nodes in the set of nodes, wherein the second indication includes a second identifier of the second node; and In response to the second indication, one of the corresponding framework daemons successfully disassociates the second node from any execution unit included in the execution context.

3. The system of claim 2, wherein: Execution of the instructions further causes the one or more processors to: An extension request including a replacement request is sent to a workload manager by the one of the corresponding framework daemons that successfully disassociates the second node from any execution unit included in the execution context.

4. The system of claim 1, wherein: Execution of the workload is complete when each node in the set of nodes has entered a barrier state and each execution unit in the plurality of execution units has been executed by the set of nodes.

5. The system of claim 1, wherein: The workload execution framework further includes a framework monitor, and execution of the instructions further causes the one or more processors to: monitoring, by the framework monitor, execution of the workload on the set of nodes; and Based on the monitoring, the number of nodes included in the set of nodes is adjusted by sending an expansion request to a workload manager.

6. The system of claim 5, wherein: The extension request includes an add request.

7. The system of claim 5, wherein: The extension request includes a removal request.

8. A computer-implemented method comprising: receiving, at a framework daemon of a workload execution framework, an indication to begin executing a workload, the framework daemon executing on a first node of a set of nodes assigned by a workload manager to execute the workload, wherein the workload is divided into a plurality of execution units; In response to the indication, accessing, by the framework daemon process, an execution context including the plurality of execution units; Declaring, by the framework daemon process, that the first execution unit is executed by the first node by associating a first identifier of the first node with a first execution unit among the plurality of execution units; executing the first execution unit by the first node; After the first node completes execution of the first execution unit, the framework daemon process determines, by accessing the execution context, whether the execution context includes a second execution unit that is not declared in the plurality of execution units; When it is determined that the execution context includes the undeclared second execution unit, declaring, by the framework daemon process, execution of the second execution unit by the first node by associating the first identifier of the first node with the second execution unit; and The second execution unit is executed by the first node.

9. The computer-implemented method of claim 8, further comprising: determining, by a framework monitor of the workload execution framework, that a second node in the set of nodes that is executing the workload has failed; In response to determining that the second node has failed, the framework monitor sends a second indication that the second node has failed to the framework daemons corresponding to the non-failed nodes in the group of nodes, wherein the second indication includes a second identifier of the second node; as well as In response to the second indication, one of the corresponding framework daemons successfully disassociates the second node from any execution unit included in the execution context.

10. The computer-implemented method of claim 9, further comprising An extension request including a replacement request is sent to a workload manager by the one of the corresponding framework daemons that successfully disassociates the second node from any execution unit included in the execution context.

11. The computer-implemented method of claim 8, wherein: Execution of the workload is complete when each node in the set of nodes has entered a barrier state and each execution unit in the plurality of execution units has been executed by the set of nodes.

12. The computer-implemented method of claim 8, wherein: The workload execution framework further includes a framework monitor, and the computer-implemented method further includes: monitoring, by the framework monitor, execution of the workload on the set of nodes; and Based on the monitoring, the number of nodes included in the set of nodes is adjusted by sending an expansion request to a workload manager.

13. The computer-implemented method of claim 12, wherein: The extension request includes an add request.

14. The computer-implemented method of claim 12, wherein: The extension request includes a removal request.

15. A non-transitory computer-readable medium storing a program for execution by one or more processors, the program comprising instructions for performing the following operations: An indication to begin executing a workload is received at a framework daemon of a workload execution framework, the framework daemon executing on a first node of a set of nodes assigned by a workload manager to execute the workload, wherein The workload is divided into a plurality of execution units; In response to the indication, accessing, by the framework daemon process, an execution context including the plurality of execution units; Declaring, by the framework daemon process, that the first execution unit is executed by the first node by associating a first identifier of the first node with a first execution unit among the plurality of execution units; executing the first execution unit by the first node; After the first node completes execution of the first execution unit, the framework daemon process determines, by accessing the execution context, whether the execution context includes a second execution unit that is not declared among the plurality of execution units; When it is determined that the execution context includes the undeclared second execution unit, declaring, by the framework daemon process, execution of the second execution unit by the first node by associating the first identifier of the first node with the second execution unit; and The second execution unit is executed by the first node.

16. The non-transitory computer readable medium of claim 15, wherein: The program includes further instructions for: determining, by a framework monitor of the workload execution framework, that a second node in the set of nodes that is executing the workload has failed; In response to determining that the second node has failed, the framework monitor sends a second indication that the second node has failed to the framework daemons corresponding to the non-failed nodes in the set of nodes, wherein the second indication includes a second identifier of the second node; and In response to the second indication, one of the corresponding framework daemons successfully disassociates the second node from any execution unit included in the execution context.

17. The non-transitory computer readable medium of claim 16, wherein: The program includes further instructions for: An extension request including a replacement request is sent to a workload manager by the one of the corresponding framework daemons that successfully disassociates the second node from any execution unit included in the execution context.

18. The non-transitory computer readable medium of claim 15, wherein: Execution of the workload is complete when each node in the set of nodes has entered a barrier state and each execution unit in the plurality of execution units has been executed by the set of nodes.

19. The non-transitory computer readable medium of claim 15, wherein: The program includes further instructions for: monitoring, by a framework monitor, execution of the workload on the set of nodes; and Based on the monitoring, the number of nodes included in the set of nodes is adjusted by sending an expansion request to a workload manager.

20. The non-transitory computer readable medium of claim 19, wherein: The extension request includes at least one of the following: Addition requests; and Removal request.

Images