AUTO-SCALING, RESILIENT, AND LOAD BALANCING FRAMEWORK FOR WORKLOAD DEPLOYMENT IN CLOUD AND HIGH-PERFORMANCE COMPUTING ENVIRONMENTS

DE102024137248A1Pending Publication Date: 2025-09-18HEWLETT PACKARD ENTERPRISE DEV LP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE102024137248
Authority / Receiving Office
DE · DE
Patent Type
Applications
Current Assignee / Owner
Priority Date
2024-03-18
Filing Date
2024-12-11
Publication Date
2025-09-18

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

In certain embodiments, a method comprises receiving an indication at a framework daemon to begin executing a workload divided into execution units; accessing, by the framework daemon and in response to the indication, an execution context containing the execution units; associating, by the framework daemon, an identifier of a node with an execution unit of the execution units to claim the execution unit for execution by the node; executing the execution unit by the node; accessing, by the framework daemon, the execution context and, after the node has completed executing the execution unit, determining whether the execution context contains an unclaimed execution unit;Associating, by the framework daemon, the node's identifier with the unclaimed execution unit and, if the execution context contains an unclaimed execution unit, claiming the unclaimed execution unit for execution by the node; and executing the execution unit by the node.
Need to check novelty before this filing date? Find Prior Art

Description

GOVERNMENT STATEMENT OF SUPPORT

[0001] This invention was made with government support under contract number H98230-15-D-0022 / 0003, awarded by the Maryland Procurement Office. The government has certain rights in this invention. BACKGROUND

[0002] In data processing environments, workloads often run across more than one node. In such scenarios, a workload manager can allocate a number of nodes to run the workload and assign portions of the workload to each of the assigned nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Certain examples discussed herein are described with reference to the accompanying drawings listed below. However, the accompanying drawings only illustrate certain aspects or implementations of the examples described herein and are not intended to limit the scope of the claims. Fig. 1 shows a block diagram of an example system for implementing a workload execution framework according to one or more examples disclosed herein; Fig. 2 shows a block diagram of an example execution context 116 in accordance with one or more examples disclosed herein; Fig. 3 shows an overview of an example method for executing a workload in an HPC environment using a workload execution framework according to one or more examples disclosed herein; Fig. 4 shows an overview of an example method for executing a workload in an HPC environment using a workload execution framework in the event of a node failure according to one or more examples disclosed herein; and Fig. 5 shows a block diagram of a computing device in accordance with one or more examples disclosed herein; Fig. 6 shows a block diagram of a computing device in accordance with one or more examples disclosed herein. DETAILED DESCRIPTION OF EXAMPLES FOR ILLUSTRATIVE PURPOSES

[0004] High-performance computing (HPC) workloads, workloads running in a cloud computing environment, and many other types of workloads often include any number of workload parts (e.g., predetermined portions of the overall workload) that can be executed on any number of nodes (e.g., compute devices) deployed in an HPC environment (e.g., a data center, a cloud environment, or another type of HPC environment). For example, an HPC environment may include a large number of heterogeneous nodes (e.g., thousands, tens of thousands) that may have different capabilities with respect to various resources (e.g., compute resources, network resources, accelerator resources (e.g., graphics processing units (GPUs)), storage resources, any other type of accelerator). In such an HPC environment, the execution of a workload may require a scheduler, workload manager, or the like that allocates resources (e.g.,Nodes) and distributes the various workload parts across all or part of the nodes of the HPC environment. A workload can be any set of operations, actions, processes, or other activities performed in an HPC environment by the nodes within it.

[0005] However, in an HPC environment, certain issues can arise that prevent and / or reduce the efficient execution of workloads. One such issue is that different portions of the workload to be executed are often statically pre-assigned to the nodes of the HPC environment. Such static assignments often do not take into account the different capabilities of the nodes in an HPC environment and / or the different rates at which the execution of workload portions can complete. Such static assignment of workload portions can result in a portion of the nodes that have completed their statically assigned workload portion having to wait for the other nodes to complete their statically assigned workload portions before the workload execution can continue and / or complete.

[0006] Another problem arises with node failures. Node failures are often addressed with some kind of checkpoint-and-restart mechanism, which requires periodic checks of the workload's progress (which incurs time and resource overhead) and the workload to be restarted from the last checkpoint if a node in the group of nodes running the workload fails.

[0007] Another problem that may arise related to the execution of workloads by nodes in an HPC environment is that a workload may be divided into a certain number of workload parts and the execution of the workload may be delayed while the required number of nodes sufficient to execute the workload parts becomes available in the HPC environment.

[0008] To address at least the above-noted problems, the examples disclosed herein include techniques for providing a workload execution framework for workload execution in HPC environments that is capable of self-scaling, resiliency, and load balancing. In one or more examples, the workload execution framework includes: a framework monitor, which may be included, for example, in a workload manager of the HPC environment; a framework daemon running on each node of an HPC environment used to execute a workload; and a framework execution context, which may be included in a shared memory / storage device accessible by each of the nodes and that stores information regarding execution units (e.g., workload chunks) into which a workload has been divided prior to execution.

[0009] In one or more examples, before executing a workload in an HPC environment, the workload may be divided into any number of execution units. In one or more examples, an execution unit is any portion of a workload that can be executed by a node. For example, for a workload that involves multiplying a sparse matrix, an execution unit may comprise one of the matrix rows to be multiplied. In one or more examples, an execution unit is uniquely identified in shared memory within the set of execution units. For example, each execution unit may be numbered, have a unique identifier, etc.

[0010] In one or more examples, the execution context may be stored in any form of computer memory accessible by the nodes of an HPC environment that execute the workload by executing the execution units, such as a fabric-attached storage device accessible by any node, a shared memory pool accessible by any node, a network-attached storage device accessible by any node, etc. The execution context may be stored in any other form of computer memory without exceeding the scope of the examples disclosed herein. The execution context may include a listing of the execution units along with the execution unit identifiers.The execution context may contain any other information relevant to the execution units without departing from the scope of the examples disclosed here. For example, the execution context may contain a table listing a set of numbered execution units and including a field for the identifier of a node executing the execution unit (see below).

[0011] When a workload is to be executed in an HPC environment, in one or more examples, a workload manager may be configured to allocate at least a portion of the nodes of the HPC environment to execute the workload. However, in one or more examples, the execution units into which a workload is divided are not statically assigned to the nodes. Instead, in one or more examples, the nodes allocated for workload execution are configured to claim execution units, execute the claimed execution units, and then claim any remaining unexecuting execution units, where any node can claim any execution unit.In one or more examples, such dynamic claiming of execution units provides nodes with the flexibility to continue workload execution even in the event of a node failure (discussed below) and to scale nodes as needed (e.g., increase the number of nodes, decrease the number of nodes, replace failed nodes). The number of nodes allocated to execute the workload may depend on any number of factors, such as the number of nodes available to execute the workload at a given time, the number of execution units into which the workload was pre-divided, service level agreement parameters such as execution time, or any other relevant factor.Other factors may affect the number of nodes allocated by a workload manager to execute a workload, without exceeding the scope of the examples disclosed here.

[0012] The number of nodes allocated to execute a workload may be equal to or less than the number of execution units. For example, a workload may be divided into 30 execution units, and a workload manager may allocate 10 nodes to execute the workload based on a desired workload execution time. In one or more examples, a workload manager may allocate fewer nodes to a workload due to a lack of node availability to get the workload started, and increase the number of nodes allocated to the workload as the workload runs and more nodes become available (e.g., to auto-scale to meet workload execution requirements without preventing workload execution from starting while waiting for more nodes to become available).

[0013] In one or more examples, after the predetermined execution units of a workload have been loaded into the execution context and a set of nodes (e.g., one or more) have been assigned by a workload manager to execute the workload, the nodes assigned to execute the workload may be given an indication (e.g., a signal) to begin executing the workload.

[0014] In one or more examples, in response to such an indication, the nodes may execute a framework daemon (e.g., using an application programming interface (API)) that accesses the execution context. Each node's framework daemon may access the execution context, select one or more execution units to be executed by the respective node on which the framework daemon is executing, and provide an identifier of the node associated with the one or more execution units selected for execution by the node.

[0015] For example, for a particular node, the framework daemon may select an execution unit for execution by the node and populate a field in the execution context associated with the execution unit with an identifier of the node. The node's identifier may be any form of information that uniquely identifies the node among the set of nodes executing the workload (e.g., a node number, a serial number, or other identifier that is unique among the nodes). In one or more examples, associating a node's identifier with one or more execution units indicates to each of the other nodes in the execution context that the node is executing the one or more execution units that the framework daemon has associated with the node's identifier.

[0016] In one or more examples, in the absence of other factors (e.g., node failure, scaling requirements, see below), execution of the workload continues by each of the nodes assigned to execute the workload claiming at least one execution unit in the execution context and executing the execution unit. In one or more examples, once a node completes the execution of its claimed execution unit, the node's framework daemon reaccesses the execution context to determine if there are any additional execution units within it that have not yet been claimed for execution.In one or more examples, if there are one or more unclaimed execution units, the node's framework daemon may claim one or more of the unclaimed execution units for execution by the node by associating an identifier of the node with the one or more execution units, as described above, to indicate to the other nodes that the node is executing the one or more execution units.

[0017] In one or more examples, the node may enter a blocking state when a node's framework daemon determines that there are no more unclaimed execution units in the execution context. In one or more examples, once all nodes have entered the blocking state, the execution of the workload assigned to the nodes may be considered complete.

[0018] In one or more examples, the fact that nodes request execution units when they complete the execution of previous workload units provides dynamic load balancing for workload execution, as nodes that can execute execution units faster can execute additional execution units of the workload. For example, a group of nodes may be heterogeneous, with some nodes having more resources (e.g., GPUs, memory resources). Thus, some nodes may execute execution units of a workload faster than other nodes assigned to execute the workload.In such scenarios, instead of completing a statically allocated portion of the workload and then waiting while other nodes complete their respective statically allocated portions of the workload, a node with more resources that completes an execution unit can dynamically request additional execution units to run, thereby executing a larger portion of the workload than the nodes with fewer resources, thus implicitly balancing the load between nodes.

[0019] In one or more examples, one or more nodes from the set of nodes assigned by a workload manager to execute a workload may fail. In one or more examples, the framework monitor of the workload execution framework may monitor the nodes assigned to execute the workload. For example, the framework monitor may run as part of the workload manager, may be a separate compute device operatively connected to the workload manager, and / or may be part of a separate compute device operatively connected to the framework manager. The framework monitor may monitor the nodes of the framework to determine whether a node has failed, among other possible monitoring tasks (see below).

[0020] For example, the framework daemon running on each node may be configured to send a periodic heartbeat signal to the framework monitor while the workload is running. In one or more examples, the framework monitor may determine that the node has failed if the heartbeat signal is not received from a node at an appropriate time. In one or more examples, the framework monitoring service may be configured so that, in the event of a node failure, it may communicate a signal to the framework daemons running on each of the other nodes indicating that the failed node has failed.

[0021] In one or more examples, the various framework daemons may respond to the framework monitor's indication by accessing the execution context and attempting to remove the failed node's identifier from each execution unit to which the failed node was connected and thus executing. In one or more examples, each node's framework daemon may attempt such an action, but only one will succeed because, after the first framework daemon succeeds, the remaining framework daemons will not find a failed node identifier associated with any execution unit in the execution context.In one or more examples, removing the identifier linking the failed node to an execution unit results in the execution unit being returned to a state where it is available for execution by one of the remaining non-failed nodes (e.g., as an unclaimed execution unit). Therefore, once one of the nodes completes the execution unit currently being executed by that node, the node can access the execution context and see the execution unit previously executed by the now-failed node as an unclaimed execution unit and request execution for execution, as described above. This allows the workload to continue executing even in the event of a node failure, ensuring the resilience of the workload execution.

[0022] In one or more examples, the framework daemon of a node that has successfully broken the link between an execution unit and the failed node in the execution context may be further configured to request resources to replace the failed node (e.g., another node) from the workload manager that allocated the nodes to execute the workload. If, as described above, no additional nodes are available, execution of the workload may be resumed by the remaining non-failed nodes, which continue to request and execute execution units from the execution context. However, if resources (e.g.,If two or more nodes (e.g., another node) are available to replace the failed node, the workload manager can respond to the framework daemon's request by adding another node to the set of nodes running the workload to replace the failed node, thereby providing an additional level of resilience and error recovery for workload execution. In one or more examples, the new node added to the set of nodes is configured with a framework daemon (like the other nodes), accesses the execution context, and begins the process of requesting execution units for execution described above.

[0023] In one or more examples, the framework monitor may be configured to monitor the execution of the workload by the set of nodes assigned by the workload manager to execute the workload. The framework monitor may monitor any aspect of the workload's execution, including, but not limited to, how long execution units wait before executing, contention for resources among the set of nodes executing the workload, the availability of resources in the HPC environment, the estimated execution time, and / or any other aspect of the workload's execution by the nodes. Based on such monitoring, the framework monitor may make scaling requests to the workload manager, which allocates resources to the workloads in the HPC environment.

[0024] For example, a framework monitor may monitor the execution of a workload and determine that the execution rate of the workload's execution units may cause the execution time for the workload to exceed the desired or scheduled execution time, and in response, request the workload manager to add additional nodes to the set of nodes executing the workload, thereby increasing the execution speed for the workload, which may be referred to as a boost request.

[0025] As another example, the framework monitor may detect that one or more nodes from the set of nodes assigned to execute a workload have no more execution units and are thus idle, and in response, send a message to the workload manager that the one or more nodes are no longer assigned to the workload and can thus be made available to execute other workloads, which may be referred to as an eviction request.

[0026] In one or more examples, the Framework Monitor's monitoring of the workload execution on the nodes assigned to the workload may enable automatic scaling of resource utilization in the HPC environment by adding and / or removing nodes as needed to meet the workload execution goals without leaving unused nodes idle.

[0027] Certain examples of this disclosure may address specific issues associated with workload execution in an HPC environment due to the static allocation of workload portions to resources of the HPC environment, such as the lack of dynamic load balancing between nodes after execution has begun, the inability of workload managers to handle scenarios where resources fail, and the inability to scale resources to meet workload execution demands without leaving allocated resources unused.

[0028] These and other problems can be addressed by the examples disclosed here, which provide a workload execution framework that enables nodes tasked with executing a workload to request execution units into which the workload is divided to be executed by the nodes, request additional execution units after the execution of previous execution units has completed, detach execution units from failed nodes so that they can be re-executed by other nodes, request node replacement from a workload manager in the event of a node failure, and automatically scale resources of an HPC environment to meet workload execution demands by monitoring workload execution through a framework monitor.

[0029] Fig. 1 shows a block diagram of an example system for implementing a workload execution framework according to one or more examples of this disclosure. The system may include a high-performance computing (HPC) environment 100. The HPC environment 100 may include a workload manager 102 and any number of nodes 104 (in Fig. 1 as Node A 106, Node B 108, and Node N 110). The HPC environment may also include a workload execution framework 112. The workload execution framework 112 may include an execution context 116, a framework monitor 114 (which may be included as part of the workload manager 102), and any number of framework daemons (e.g., Framework Daemon A 118, Framework Daemon B 120, Framework Daemon N 122) executing on corresponding nodes (e.g., Node A 106, Node B 108, Node N 110). Each of these components is described below.

[0030] In one or more examples, HPC environment 100 is any collection of resources for executing workloads. Thus, HPC environment 100 may include any number of compute resources (e.g., any number of nodes such as 106, 108, 110, totaling 104 nodes), any number of network resources (not shown), and any number of storage resources (not shown), as well as any number of other resources, devices, components, and the like (e.g., workload manager 102) that facilitate the execution of workloads.

[0031] The HPC environment 100 may be deployed in a cloud computing environment (e.g., public cloud, private cloud, hybrid cloud), a data center environment, or any other computing environment with the aforementioned resources. The resources of the HPC environment 100 may be located at the same physical location or distributed across any number of separate physical locations, which may or may not change over time.

[0032] In one or more examples, the HPC environment 100 includes any number of nodes 104 (e.g., Node A 106, Node B 108, Node N 110). In one or more examples, a node (e.g., Node A 106, Node B 108, Node N 110) is a computing device. In one or more examples, as used herein, a computing device, such as one of nodes 106, 108, and 110, may be a single computing device, a set of computing devices, a portion of one or more computing devices, or other physical, virtual, and / or logical grouping of computing resources. An example of a computing device is described in Fig. 5 and is described below.

[0033] Examples of computing devices include, but are not limited to, a server (e.g., a blade server in a blade server enclosure, a rack server in a rack, a desktop server, any other type of server device), a desktop computer, a mobile device (e.g., a laptop computer, a smartphone, a personal digital assistant, a tablet computer, a computer system in a car, and / or any other mobile computing device), a storage device (e.g., a hard disk drive array, a Fibre Channel storage device, an Internet Small Computer Systems Interface (iSCSI) storage device, a tape storage device, a flash memory array, a network-attached storage device, any other type of storage device), a network device (e.g., a switch, router, multi-layer switch, any other type of network device), a virtual machine, a virtualized computing environment, a logical container (e.g.,for one or more applications), a container pod, an Internet of Things (IoT) device, an arrangement of nodes of computing resources, a supercomputing device, a data center or part thereof, and / or any other type of computing device with the above requirements.

[0034] In one or more examples, some or all of the above examples may be combined to create a system of such devices, or they may be separated into separate logical devices that may collectively be referred to as a computing device. Other types of computing devices may also be used without departing from the scope of the examples described herein, such as the Fig. 5 and described below. The HPC environment 100 may include any number and / or type of such nodes (e.g., Node A 106, Node B 108, Node N 110) in any arrangement and / or configuration without departing from the scope of the examples disclosed herein.

[0035] In one or more examples, the storage and / or memory of a computing device or system of computing devices (e.g., Node A 106, Node B 108, Node N 110) may be and / or include one or more data stores to store any number of data structures that store any amount of data (e.g., information). In one or more examples, a data store is any type of storage unit and / or device (e.g., a file system, a database, a spreadsheet, RAM, a hard disk drive, a solid-state drive, and / or other storage mechanism or storage medium) for storing data. Furthermore, the data repository may include multiple different storage units and / or devices. The different storage units and / or devices may be of the same type or located in the same physical location.

[0036] In one or more examples, any memory and / or storage of a computing device or system of computing devices and / or network devices may be considered, in whole or in part, to be a non-transitory computer-readable medium storing software and / or firmware that, when executed by one or more processors, causes the one or more processors to perform operations in accordance with one or more examples disclosed herein.

[0037] The HPC environment 100 may include any number of nodes 104, which may be individually or collectively considered a computing device (as used herein). All or some of the computing devices may be of the same type or different types of computing devices.

[0038] In one or more examples, HPC environment 100 includes workload manager 102. In one or more examples, workload manager 102 is, in whole or in part, a computing device (described above). In one or more examples, workload manager 102 is configured to manage, at least in part, the deployment and execution of workloads within HPC environment 100. In one or more examples, workload manager 102 is configured to allocate any number of nodes (e.g., Node A 106, Node B 108, Node N 110) for execution of a particular workload and to configure, set up, provision, or otherwise make available other portions of HPC environment 100 (e.g., network resources, storage resources, any other type of resources) that may be required to execute a workload.

[0039] In one or more examples, the workload manager is operatively connected to each of the nodes 104 (e.g., Node A 106, Node B 108, Node N 110) of the HPC environment 100. In one or more examples, the workload manager 102 may be configured such that, when a workload is to be executed in the HPC environment 100, the workload manager 102 allocates a number of nodes (e.g., nodes 104) to execute the workload. The number of nodes allocated to execute the workload may depend on any number of factors, such as the number of nodes available to execute the workload at a given time, the number of execution units into which the workload was pre-divided, service level agreement parameters such as execution time, and / or other relevant factors.Other factors may affect the number of nodes allocated by workload manager 102 to execute a workload without departing from the scope of the examples disclosed herein.

[0040] In one or more examples, the workload execution framework 112 is deployed within the HPC environment 100. In one or more examples, the workload execution framework 112 is a collection of components that includes the framework monitor 114, any number of framework daemons (e.g., 118, 120, 122), and the execution context 116, each of which is described below. As in Fig. 1, the workload execution framework 112 is represented by the dashed box, which includes the aforementioned collection of components, including the framework monitor 114, any number of framework daemons (e.g., 118, 120, 122), and the execution context 116. While the workload execution framework 112 includes the aforementioned components, it may not include all devices that include such components. For example, the workload execution framework 112 may include the framework monitor 114, but not the workload manager 102, which includes the framework monitor 114 beyond the framework monitor 114. As another example, the workload execution framework 112 may include the framework daemons 118, 120, and 122, but not the rest of their respective nodes 106, 106, 122 outside of the framework monitors.In one or more examples, workload execution framework 112 facilitates the execution of workloads according to the examples disclosed herein. As such, the various components of workload execution framework 112 cooperate with each other to enable workload execution that is resilient (e.g., capable of handling node failures), load-balanced, and automatically scalable.

[0041] In one or more examples, prior to executing a workload in the HPC environment 100 using the workload execution framework 112, the workload may be divided into any number of execution units. In one or more examples, an execution unit is any portion of a workload that may be executed by a node (e.g., Node A 106, Node B 108, Node N 110) of the HPC environment 100. For example, for a workload that includes executing a series of operations as separate processes, an execution unit may include one of the processes. In one or more examples, an execution unit is uniquely identified within the set of execution units.For example, each execution unit may be numbered, have a unique identifier, or be uniquely identified within the set of execution units by another suitable technique for distinguishing an element from other elements in a set of elements. In one or more examples, the execution units into which the workload is divided are not statically assigned to nodes prior to execution of the workload.

[0042] In one or more examples, the workload execution framework 112 includes the execution context 116. In one or more examples, the execution context 116 is any component configured to store information and operatively connected to at least the nodes 104 allocated by the workload manager 102 for execution of a workload.In one or more examples, the execution context may be any form of storage accessible by the nodes 104 of the HPC environment 100 that execute the workload by executing the execution units, such as a fabric-attached storage device accessible by any node, a shared storage pool accessible by any node, a network-attached storage device accessible by any node, and / or any other form of persistent or non-persistent storage or memory accessible by any of the nodes 104.

[0043] In one or more examples, the execution context 116 is configured to store information related to the aforementioned execution units into which a workload has been divided (but which has not been pre-assigned to any node prior to execution of the workload by the nodes 104 of the HPC environment 100). The execution context 116 may include a listing of the execution units along with the identifiers of the execution units, or a listing of the identifiers. The execution context may include any other information relevant to the execution units without exceeding the scope of the examples disclosed herein. For example, the execution context may include a table listing a number of numbered execution units and including a field for an identifier of a node executing the execution unit (see below).An example of the execution context 116 is described below in the description of . Fig. 2 discussed.

[0044] In one or more examples, the workload execution framework 112 includes any number of framework daemons (e.g., 118, 120, 122). In one or more examples, a framework daemon is any hardware, software, and / or firmware configured on, or part of, and / or executing on one of the nodes 104 assigned by the workload manager 102 to execute a workload. In one or more examples, a framework daemon (e.g., 118, 120, 122) is configured to use one or more APIs to interact with other components of the HPC environment 100 and the workload execution framework 112, such as the execution context 116, the framework monitor 114, and / or the workload manager 102.

[0045] In one or more examples, the framework daemons (e.g., 118, 120, 122) are configured to receive an indication that workload execution is about to begin and, in response to such an indication, to access the execution context 116 to request one or more execution units to be executed by the respective node on which the framework daemon is deployed. The operations and actions performed by a framework daemon (e.g., 118, 120, 122) are described further below in the descriptions of Fig. 3 and Fig. 4 discussed.

[0046] In one or more examples, the workload execution framework 112 includes the framework monitor 114. In one or more examples, the framework monitor 114 consists of all or any part of a computing device (described above). For example, the framework monitor 114 may be a part of the workload manager 102 (e.g., as described in Fig. 1). Alternatively, the framework monitor 114 may be separate from and operatively connected to the workload manager 102. In one or more examples, the framework monitor 114 is operatively connected to each of the nodes 104 and, thus, to each of the framework daemons (e.g., 118, 120, 122) executing thereon.

[0047] In one or more examples, the framework monitor 114 is configured to monitor the execution of the workload on the nodes 104. In one or more examples, the framework monitor 114 may be configured to monitor the nodes 104 for any node failures. For example, the framework daemon (e.g., 118, 120, 122) running on each node may be configured to send a periodic heartbeat signal to the framework monitor 114 while the workload is executing. In one or more examples, the framework monitor 114 may determine that the node has failed if the heartbeat signal is not received from a node at an appropriate time. The monitoring of node failures by the framework monitor 114 is described further below in the description of Fig. 4 discussed.

[0048] In one or more examples, the framework monitor 114 may be configured to monitor the execution of the workload by the group of nodes 104 assigned by the workload manager 102 to execute the workload. The framework monitor 114 may monitor any aspect of the execution of the workload, including, but not limited to, how long execution units wait before executing, contention for resources among the group of nodes 104 executing the workload, availability of resources in the HPC environment 100, estimated execution time, and / or any other relevant aspect of the workload execution on the nodes. Based on such monitoring, the framework monitor 114 may make scaling requests to the workload manager 102, which allocates resources to the workloads in the HPC environment 100.

[0049] For example, the framework monitor 114 may monitor the execution of a workload and determine that the execution rate of the workload's execution units may cause the execution time for the workload to exceed the desired or scheduled execution time, and in response, issue a scaling request to the workload manager 102 to add additional nodes to the set of nodes 104 executing the workload (which may be referred to as a boost request), thereby increasing the execution rate for the workload.

[0050] As another example, the framework monitor 114 may determine that one or more nodes in the group of nodes 104 assigned to execute a workload have no more execution units and are thus idle, and in response, send an indication to the workload manager 102 that the one or more nodes may be released from assignment to the workload (which may be referred to as an eviction request) and thus made available to execute other workloads.

[0051] Thus, in one or more examples, monitoring of the workload execution on the nodes 104 assigned to the workload by the framework monitor 114 may enable automatic scaling of resource utilization in the HPC environment 100 by adding and / or removing nodes as needed to meet the workload execution goals without leaving unused nodes unused.

[0052] Although Fig. 1 shows a specific configuration of components, other configurations may be used without deviating from the scope of the examples described here. Although in Fig. 1 certain components are shown as part of one and the same device, each of the components may be grouped into sets of one or more components that may exist and execute as part of any number of separate and operatively connected devices. Another example is that a single component may be configured to provide all or part of the functionality of all or part of the devices shown in Fig. 1. As a further example, only certain components of the HPC environment 100 are shown for clarity. However, the HPC environment 100 may include any number of additional components (e.g., network devices, storage devices, management devices, etc.) without departing from the scope of the examples disclosed herein. Accordingly, the examples disclosed herein should not be limited to the configuration of the components disclosed in Fig. 1 shown components.

[0053] Fig. 2 shows a block diagram of an exemplary execution context 216 according to one or more examples disclosed herein. As in Fig. 2, execution context 216 includes a representation of any number of execution units (e.g., execution unit A 200, execution unit B 202, execution unit N 204) and any number of node identifiers (e.g., node identifier A 206, node identifier B 208). Each of these components is described below.

[0054] In one or more examples, execution context 216 is an example of the Fig. 1 and described above. As such, the execution context 216 is any memory or storage device configured to store information of any type in any form. The execution context may be part of a computing device (as described above) or a memory and / or storage accessible by computing devices (e.g., the memory devices described in Fig. Node 104 shown in Figure 1).

[0055] In one or more examples, the execution context 216 is configured to at least temporarily store a data structure of any type capable of linking information elements together. In one or more examples, the data structure stored in the execution context 216 is configured to support the allocation of execution units of a workload to nodes (e.g., the nodes in Fig. 1) claiming the execution units. In one or more examples, the nodes claim an execution unit by executing a framework daemon (e.g., framework daemon A 118, framework daemon B 120, framework daemon N 122, shown in Fig. 1) that accesses the execution context 216 to find execution units represented therein and claims an execution unit for execution on the framework daemon's node. In one or more examples, a framework daemon may claim an execution unit by associating a node identifier (e.g., 206, 208) of the node on which the framework daemon is executing with an execution unit represented in the execution context 216.

[0056] For example, for a particular node, the framework daemon may select an execution unit for execution by the node and populate a field associated with the execution unit (e.g., Execution Unit A 200, Execution Unit B 202, Execution Unit N 204) in the execution context with an identifier of the node (e.g., Node Identifier A 206, Node Identifier B 208). The identifier of the node (e.g., Node Identifier A 206, Node Identifier B 208) may be any form of information that uniquely identifies the node within the group of nodes executing the workload (e.g., a node number, a serial number). In one or more examples, associating a node's identifier with one or more execution units indicates to each of the other nodes in the execution context 216 that the node is executing the one or more execution units that the framework daemon has associated with the node's identifier.

[0057] In one or more examples, a portion of the execution units in execution context 216 may be unclaimed, such as those in Fig. 2. In one or more examples, an execution unit may be unclaimed in a variety of scenarios. For example, if a workload is divided into more execution units than nodes are assigned to execute the workload, after each node claims an execution unit, a portion of the execution units remain as unclaimed execution units, which are claimed by the nodes when the nodes complete the execution of the previously claimed execution units. As another example, an execution unit that was executing by a failed node may return to an unclaimed state (see the description of Fig. 4).

[0058] In one or more examples shown in Fig. 2, execution context 1216 may be configured to indicate in the data structure of the execution units and the associated nodes executing the execution units that an execution unit has already executed, and / or to remove the representation of the execution units from execution context 216 when the execution unit has been completed by a node. In one or more examples, the execution of a workload may be considered complete when all execution units originally included in execution context 216 have been executed by the nodes tasked with executing the workload.

[0059] Fig. 3 shows an overview of an example method 300 for executing a workload in an HPC environment using a workload execution framework according to one or more examples disclosed herein. The method 300 may be implemented at least in part by a workload execution framework (e.g., the one described in Fig. 1) or any component of a workload execution framework (e.g., the framework monitor 114 of Fig. 1, any framework daemon (e.g. 118, 120, 122) from Fig. 1, the execution context 116 of Fig. 1) and / or the workload manager 102 of Fig. 1 should be carried out.

[0060] While the various steps in the Fig. 3 are shown and described sequentially, some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, and some or all of the steps may be performed in parallel with other steps of Fig. 3 and / or Fig. 4 are executed.

[0061] In step 302, the method 300 includes receiving an indication of the start of execution of the workload at a framework daemon (e.g., the framework daemons 118, 120, 122 of Fig. 1) a workload execution framework (e.g., the workload execution framework 112 of Fig. 1) executing on a first node of a group of nodes assigned by a workload manager to execute a workload. In one or more examples, the workload is divided into a plurality of execution units prior to execution of the workload, and the execution units are not statically assigned to the nodes until the nodes' respective framework daemons receive an indication to begin workload execution. In one or more examples, the indication is received from any source that can provide an indication of the start of workload execution (e.g., a workload manager, a framework monitor, a user device, and / or another source). In one or more examples, such an indication is received from each framework daemon of each node designated to execute the workload.The notification can take any form that signals the nodes that workload execution is about to begin. For example, the notification can be sent as one or more network packets over a network to the framework daemons of the nodes that are to execute the workload.

[0062] In step 304, the method 300 includes accessing an execution context (e.g., the execution context 116 of Fig. 1, the execution context 216 of Fig. 2) containing representations of the plurality of execution units, by the framework daemon and in response to the display. For example, the framework daemon may access the execution context via an operational network connection between the node on which the framework daemon is executing and the execution context. In one or more examples, the execution context is as described above in the descriptions of Fig. 1 and Fig. 2, any memory and / or storage operatively connected to the set of nodes assigned to execute the workload. In one or more examples, the execution context comprises a data structure containing a representation of the execution units into which the workload is divided prior to commencing execution of the workload. In one or more examples, the data structure is configured to contain a field associated with each execution unit of the workload, which may store an identifier of a node, which may result in the execution unit being deemed to have been executed by the node identified by the node identifier.

[0063] In step 306, method 300 includes associating, by the framework daemon, a first identifier of the first node with a first execution unit from the plurality of execution units in the execution context to claim the first execution unit for execution by the first node. In one or more examples, a framework daemon may associate the node on which the framework daemon is executing with an execution unit by populating the field associated with the execution unit in the execution context with an identifier of the node. An identifier of a node may be any information that uniquely identifies a node from a set of nodes assigned (e.g., by a workload manager) for execution of a workload.

[0064] In step 308, method 300 includes executing the first execution unit by the first node. In one or more examples, executing an execution unit on a node includes obtaining all information necessary to execute the execution unit and then executing the execution unit. For example, if the workload is to perform sparse matrix multiplication, an execution unit may include multiplying a row of a matrix. Thus, executing the execution unit may include obtaining the row of the matrix and the element by which the row is to be multiplied, thereby enabling the node to perform the multiplication.

[0065] In step 310, the method 300 includes accessing the execution context by the framework daemon after the first node completes execution of an execution unit to determine whether the execution context includes any execution unit of the plurality of execution units that is unclaimed. In one or more examples, the execution context includes all execution units of the workload to be executed. In one or more examples, an execution unit may either be claimed by a node for execution by associating it with a node identifier, or it may be unclaimed.An execution unit cannot be claimed if no node has yet claimed the execution unit for execution, or if a node that claimed the execution unit for execution failed before the execution unit completed its execution and the execution unit was returned to an unclaimed state (as described in . Fig. 4 explained below).

[0066] In step 312, the method 300 includes determining whether the execution context contains any unclaimed execution units. In one or more examples, the determination is made by a framework daemon executing on a node that has completed executing an execution unit of a workload. In one or more examples, the determination is made by the framework daemon evaluating the execution context to determine whether any execution units contained and / or presented therein are unclaimed (e.g., not claimed by another node). In one or more examples, the method proceeds to step 318 if there are no further unclaimed execution units in the execution context. In one or more examples, the method proceeds to step 314 if there are one or more unclaimed execution units.

[0067] In step 314, the method includes associating, by the framework daemon, if it is determined in step 312 that the execution context includes the unclaimed second execution unit, the first identifier of the first node with the second execution unit to claim the second execution unit for execution by the first node. In one or more examples, a framework daemon may associate the node on which the framework daemon is executing with the second execution unit by populating the field in the execution context associated with the second execution unit with the node's identifier. The identifier of a node may be any information that uniquely identifies the node from the set of nodes assigned (e.g., by a workload manager) for execution of a workload.In one or more examples, the ability of nodes to claim additional execution units for execution after other execution units have completed execution enables implicit load balancing because nodes that have more resources and are able to complete execution units more quickly, or that have claimed an execution unit that took less time to execute, can claim additional execution units for execution.

[0068] In step 316, the method 300 includes executing the second execution unit by the first node. In one or more examples, executing a unit of execution on a node includes obtaining information necessary to execute the unit of execution and then executing the unit of execution. For example, if the workload is to execute one of a series of related processes, an execution unit may include one or more processes to be executed. Thus, executing the unit of execution may include obtaining all information necessary to execute the process and then executing the process.In one or more examples, after step 316, the method may return to step 310, where the framework daemon of a node completing execution of an execution unit accesses the execution context to determine whether any unclaimed execution units remain therein.

[0069] In step 318, the method includes entering a lock state. In one or more examples, a lock state is a state in which the nodes no longer have any execution units to execute and may be used, for example, as a form of synchronization. In one or more examples, a workload executed by a group of nodes may be considered complete when every node in the group of nodes tasked with executing the workload has reached a lock state.

[0070] Fig. 4 shows an overview of an example method 400 for executing a workload in an HPC environment using a workload execution framework in the event of a node failure, according to one or more examples disclosed herein. The method 400 may be implemented at least in part by a workload execution framework (e.g., the one described in Fig. 1) or any component of a workload execution framework (e.g., the framework monitor 114 of Fig. 1, any framework daemon (e.g. 118, 120, 122) from Fig. 1, the execution context 116 of Fig. 1) and / or the workload manager 102 of Fig. 1 should be carried out.

[0071] While the various steps in the Fig. 4 are shown and described sequentially, some or all of the steps may be performed in a different order, some or all of the steps may be combined or omitted, and some or all of the steps may be performed in parallel with other steps of Fig. 3 and / or Fig. 4 are executed.

[0072] In step 402, the method 400 includes monitoring the execution of the workload on the group of nodes (e.g., node 104 of Fig. 1) by a framework monitor (e.g. the framework monitor 114 of Fig. 1). In one or more examples, the framework monitor is configured to monitor the execution of the workload to observe various characteristics of the workload execution, including, but not limited to, the state of the nodes (e.g., down or not down). The monitoring may be performed via an operational connection between the framework monitor and each of the respective nodes.

[0073] In step 404, method 400 includes determining, by the framework monitor, whether a node has failed. In one or more examples, the framework monitor may determine that a node has failed using any suitable technique for detecting node failures. For example, framework daemons on the nodes may be configured to send a periodic heartbeat signal to the framework monitor, and failure to receive the heartbeat signal from a node at an appropriate time may indicate a node failure. As another example, the framework monitor may be configured to ping the nodes periodically, and a lack of response may indicate a node failure. As another example, the framework monitor may be configured to access the nodes periodically, and an access failure may indicate a node failure.Other methods may also be used to determine whether a node has failed without exceeding the scope of the examples disclosed herein. In one or more examples, if the framework monitor does not determine that a node has failed, the method returns to step 402, and the framework monitor continues monitoring the nodes during execution of the workload. In one or more examples, if the framework monitor determines that a node has failed, the method continues to step 406.

[0074] In step 406, the method 400 includes sending, by the framework monitor and in response to the determination that the second node has failed, an indication to the respective framework daemons of the non-failed nodes of the node set that the second node has failed. In one or more examples, the indication includes a second identifier of the second node (e.g., the failed node). The indication may be in any form suitable for notifying the framework daemons that a node has failed. In one or more examples, the indication includes at least an indication that a node has failed and an identifier of the failed node. In one or more examples, the identifier of the failed node is the same as the identifier of the node for which the failed node is configured when claiming an execution unit in the execution context.In one or more examples, the identifier of the node in the advertisement that the framework daemons receive from the framework monitor is not the same as the identifier that the failed node uses to claim execution units, but the framework daemons are configured to use the identifier received in the advertisement to determine the identifier that the failed node uses to claim execution units.

[0075] In step 408, the method includes accessing the execution context by the respective framework daemons to attempt to detach the second node from each execution unit in the execution context. In one or more examples, in response to the indication received in step 406, each of the framework daemons is configured to attempt to access the execution context and detach from the failed node any execution units contained therein that are associated with the failed node (e.g., via a link of the failed node's identifier). For example, the framework daemons may be configured to access the execution context via an API and evaluate the claimed execution units contained therein to determine whether an identifier of the failed node is associated with any of the claimed execution units.

[0076] In step 410, the method 400 includes successfully detaching the second node from each execution unit contained in the execution context by one of the respective framework daemons. In one or more examples, although each framework daemon attempts to detach the failed node from all execution units, only one will succeed because the failed node is no longer connected to all execution units after a successful attempt. In one or more examples, successfully detaching a failed node from an execution unit includes removing the identifier of the failed node from the execution context at each location in the execution context where the identifier of the failed node is associated with an execution unit. In one or more examples, detaching an execution unit from a failed node (e.g.,by removing the identifier of the failed node from the execution unit) puts the execution unit into an unclaimed state so that another node can claim the execution unit for execution.

[0077] In step 412, method 400 includes sending a scaling and replacement request to the workload manager by the one framework daemon of the respective framework daemons that successfully detached the second node from an execution unit included in the execution context. This request, if sent in response to a node failure, may be referred to as a replacement request. In one or more examples, a replacement request is a request to replace a node in the set of nodes. For example, the replacement request may be a request to replace the second node of the node set after the second node has failed.In one or more examples, one of the framework daemons is configured to attempt to detach the failed node from all execution units in the execution context in response to the indication received in step 406, and further configured, if successful as the framework daemon performing the detachment, to request a replacement node from the workload manager. In one or more examples, requiring only the successful framework daemon to send the node replacement request to the workload manager prevents the workload manager from receiving redundant requests. In one or more examples, as described in . Fig. 4 are not shown, the workload manager may respond to the request by attempting to replace the node. In one or more examples, the workload manager may not replace the node if no nodes are available in the HPC environment. In such a scenario, the workload can still continue to execute because the remaining, non-failed nodes tasked with executing the workload continue to execute execution units and then check for unclaimed execution units after the previous execution units have completed their execution (see the description of Fig. 3 above). In one or more examples, the workload manager may respond to a request to replace a node by allocating an additional node to replace the failed node in the set of nodes allocated to execute the workload, either at the time the request is received or at a later time when an additional node becomes available in the HPC environment. Thus, the examples described here provide resiliency for workload execution by either automatically replacing failed nodes or by allowing the non-failed nodes to continue workload execution even if a node fails.

[0078] Fig. 5 shows a block diagram of a computing device according to one or more examples of this disclosure. As described above, the examples described herein may be implemented using computing devices. For example, all or any portion of the Fig. 1 (e.g., the workload manager 102, the framework monitor 114, the framework daemons (e.g., 118, 120, 122), the nodes 104, the execution context 116) and the execution context 216 may be implemented at least in part using one or more computing devices, and all or any portion of the components shown in Fig. 3 and Fig. The methods shown in Figure 4 may be performed using one or more computing devices, such as computing device 500.

[0079] In one or more examples, a computing device (e.g., computing device 500) is any device, part of a device, or group of devices capable of electronically processing instructions, and may include, but is not limited to, one or more processors (e.g., components containing circuitry) (e.g., processor 502), memory (e.g., random access memory (RAM)) (not shown), input and output devices (e.g., non-persistent memory 506), non-volatile storage hardware (e.g., solid-state drives (SSDs), persistent memory devices (PMEM), hard disk drives (HDDs) (not shown)), one or more physical interfaces (e.g., network ports, storage ports) (e.g., persistent memory 506), any number of other hardware components (not shown), and / or any combination thereof.As used herein, a processor can be any component that can be configured to perform operations, processes, threads, and the like. Examples of a processor include, but are not limited to, central processing units (CPUs), multi-core CPUs, application-specific integrated circuits (ASICs), accelerators (e.g., graphics processing units (GPUs)), and field-programmable gate arrays (FPGAs).

[0080] Computing device 500 may include a communications interface 512 (e.g., a Bluetooth interface, an infrared interface, a network interface, an optical interface, or another type of communications interface), input devices 510, output devices 508, and numerous other elements (not shown) and functionalities. Each of these components is described below.

[0081] In one or more examples, the computer processor(s) 502 may be an integrated circuit for processing instructions. The computer processor(s) may be, for example, one or more cores or micro-cores of a processor. The processor 502 may be a general-purpose processor configured to execute program code included in the software executing on the computing device 500. The processor 502 may be a special-purpose processor with specific instructions integrated into the processor design. The processor 502 may be an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a data processing unit (DPU), a tensor processing unit (TPU), an association processing unit (APU), a vision processing unit (VPU), a quantum processing unit (QPU), and / or various other processing units that utilize specialized hardware (e.g.,field-programmable gate arrays (FPGAs), systems-on-a-chips (SOCs), digital signal processors (DSPs). Although in . Fig. 5 only one processor 502 is shown, the computing device 500 may include any number of processors without departing from the scope of the examples disclosed herein.

[0082] Computing device 500 may also include one or more input devices 510, such as a touchscreen, a keyboard, a mouse, a microphone, a touchpad, an electronic pen, a motion sensor, or another type of input device. Input devices 510 may enable a user to interact with computing device 500. In one or more examples, computing device 500 may include one or more output devices 508, such as a screen (e.g., a liquid crystal display (LCD), a plasma display, a touchscreen, a cathode ray tube (CRT) monitor, a projector, or other display device), a printer, external storage, or another output device. One or more of the output devices may be the same as or different from the input device(s).The input and output device(s) may be connected locally or remotely to the computer processor(s) 502, the non-persistent memory 504, and the persistent memory 506. There are many different types of computing devices, and the above-mentioned input and output device(s) may take other forms. In some cases, multimodal systems may allow a user to provide multiple types of input / output to communicate with the computing device 500.

[0083] Furthermore, the communications interface 512 may facilitate the connection of the computing device 500 to a network (not shown) (e.g., a local area network (LAN), a wide area network (WAN) such as the Internet, a mobile network, or another type of network) and / or to another device, such as another computing device. The communications interface 512 may perform or facilitate the reception and / or transmission of wired or wireless communications using wired and / or wireless transceivers of any type and / or technology.Examples include, but are not limited to, an audio jack / plug, a microphone jack / plug, a USB port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, Bluetooth® wireless signaling, BLE wireless signaling, IBEACONO wireless signaling, RFID wireless signaling, Near Field Communication (NFC) wireless signaling, Dedicated Short Range Communication (DSRC) wireless signaling, an 802.11 WiFi radio signal transmission, WLAN signal transmission, Visible Light Communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), IR communication radio signal transmission, PSTN (Public Switched Telephone Network) signal transmission, ISDN (Integrated Services Digital Network) signal transmission, 3G / 4G / 5G / LTE radio signal transmission for cellular data networks, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, radio signal transmission along the electromagnetic spectrum, or a combination thereof.The communications interface 512 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine the location of the computing device 500 based on the reception of one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, among others, the U.S. GPS, the Russian Global Navigation Satellite System (GLONASS), the Chinese BeiDou Navigation Satellite System (BDS), and the European Galileo GNSS. There is no limitation to operation on any particular hardware arrangement, and therefore the basic functions here can be easily replaced with improved hardware or firmware arrangements as they are developed.

[0084] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transmitting instructions and / or data. A computer-readable medium may be a non-transitory medium capable of storing data and which does not contain carrier waves and / or transitory electronic signals that propagate wirelessly or over wired connections. Examples of a non-transitory medium may include, but are not limited to, a magnetic disk or tape, optical storage media such as CDs or DVDs, flash memory, storage, or memory devices.A computer-readable medium may store code and / or machine-executable instructions that may represent a procedure, function, subroutine, program, routine, subprogram, module, software package, class, or any combination of instructions, data structures, or program instructions. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, and the like may be passed, relayed, or transmitted via any suitable means, including sharing memory, passing messages, passing tokens, transmitting over a network, or the like.

[0085] All or a portion of the components of computing device 500 may be implemented in circuitry. For example, the components may include electronic circuitry or other electronic hardware and / or may be implemented using electronic circuitry, which may include one or more programmable electronic circuits (e.g., microprocessors, GPUs, DSPs, FPGAs, CPUs, and / or other suitable electronic circuits), and / or may include computer software, firmware, or any combination thereof, and / or may be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some aspects, the computer-readable storage devices, media, and memories may include a wired or wireless signal containing a bitstream and the like.However, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves and signals as such.

[0086] Fig. 6 shows a block diagram of a computing device 600 according to one or more examples disclosed herein. The computing device 600 is an example of the computing devices previously described in the description of Fig. 5 described computer device 500. As described above in the descriptions of Fig. 1, Fig. 2, Fig. 3, Fig. 4 and Fig. 5, the computing device 600 may be used to perform all or any portion of the various Fig. 1 and Fig. 2 and described above, such as the workload manager 102 of Fig. 1, the framework monitor 114 of the workload manager 102 of Fig. 1, the execution context 116 of Fig. 1, node 106 of Fig. 1, the framework daemon 118 of node A 116 of Fig. 1, the node B of Fig. 1, the framework daemon 120 of node B 108 of Fig. 1, the node N 110 of Fig. 1, the framework daemon N 122 of node N 110 of Fig. 1 and / or the execution context 216 of Fig. 2 (or any component therein).

[0087] Computing device 600 may include one or more processors 602 and a memory 604. Memory 604 may include a non-transitory computer-readable medium storing programming for execution by one or more of the one or more processors 702. In this implementation, one or more modules within computing device 600 may be embodied partially or entirely as software for performing the functions described in this disclosure. For example, computing device 600 may be configured to Fig. 3 and described above by executing instructions contained in memory 604 and executed by the one or more processors 602.

[0088] For example, the memory 604 may include instructions 606 to receive, at a framework daemon of a workload execution framework executing on a first node of a set of nodes assigned by a workload manager to execute a workload, an indication to begin execution of the workload, where the workload is divided into a plurality of execution units (e.g., as described above with reference to step 302 of Fig. 3 described).

[0089] The memory 604 may also include instructions 608 for accessing, by the framework daemon and in response to the indication, an execution context comprising the plurality of execution units (e.g., as described above with reference to step 304 of Fig. 3 described).

[0090] The memory 604 may also include instructions 610 for claiming, by the framework daemon, a first execution unit from the plurality of execution units for execution by the first node by associating a first identifier of the first node with the first execution unit (e.g., as described above with reference to step 306 of Fig. 3 described).

[0091] The memory 604 may also include instructions 612 for executing the first execution unit by the first node (e.g., as described above with reference to step 308 of Fig. 3 described).

[0092] The memory 604 may also include instructions 614 to determine, by the framework daemon and after the first node has completed execution of the first execution unit, whether the execution context includes a second execution unit of the plurality of execution units that is unclaimed by accessing the execution context (e.g., as described above with reference to step 310 and step 312 of Fig. 3 described).

[0093] The memory 604 may also include instructions 616 for, by the framework daemon and if it is determined that the execution context includes the unclaimed second execution unit, to claim the second execution unit for execution by the first node by associating the first identifier of the first node with the second execution unit (e.g., as described above with reference to step 314 of Fig. 3 described).

[0094] The memory 604 may also include instructions 618 for executing the second execution unit by the first node (e.g., as described above with reference to step 316 of Fig. 3 described).

[0095] In the above description, numerous details are given as examples of the examples described herein. Those skilled in the art (and having the benefit of this disclosure) will understand that one or more examples described herein may be practiced without these specific details, and that numerous variations or modifications are possible without departing from the scope of the examples described herein. Certain details known to those skilled in the art may be omitted in order not to obscure the description.

[0096] In the above description, specific details are set forth to provide a thorough understanding of the aspects and examples presented herein. However, one skilled in the art will appreciate that the aspects and examples may be practiced without these specific details. For clarity, in some cases, the present technology may be represented as functional blocks, which may include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Components other than those shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be represented as components in block diagram form so as not to unnecessarily obscure the aspects.In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail in order not to obscure aspects of the examples disclosed herein.

[0097] Individual aspects can be described above as a process or method, which are represented in the form of a flowchart, a sequence diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be changed. A process can terminate when its operations are completed, but may also contain additional steps not included in a drawing. A process can correspond to a method, a function, a procedure, a subroutine, a subprogram, and the like. If a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0098] Methods and techniques according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from computer-readable media. Such instructions may, for example, include instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or set of functions. Portions of the computing resources used may be accessible over a network. The computer-executable instructions may, for example, be binary files, instructions in intermediate formats such as assembly language, firmware, source code, and the like.Examples of computer-readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0099] In the above description of the figures, each component described in one figure may, in various examples described herein, correspond to one or more equally or similarly named and / or numbered components described in another figure. For brevity, the descriptions of these components are not repeated for each drawing. Therefore, each individual example of the components of each figure is incorporated by reference and is assumed to be optionally present in any other figure having one or more equally or similarly named and / or numbered components.Furthermore, in accordance with various examples described herein, each description of the components of a figure is to be understood as an optional example that may be implemented in addition to, in conjunction with, or in place of the examples described with respect to one or more like or similarly named and / or numbered components in another figure.

[0100] Throughout the application, ordinal numbers (e.g., first, second, third) may be used as an adjective for an element (i.e., any noun in the application). The use of ordinal numbers is not intended to imply or create a particular order of elements, nor is it intended to limit an element to being only a single element, unless explicitly stated, such as by the use of the terms "before," "after," "single," and other such terminology. Rather, the use of ordinal numbers is intended to distinguish between elements. For example, a first element is to be distinguished from a second element, and the first element may comprise more than one element and may follow (or precede) the second element in the order of the elements.

[0101] As used herein, the term "operatively connected" or "operative link" means that a direct or indirect connection exists between elements / components / devices that enables the elements to interact with each other in some way. The term "operatively connected" can refer, for example, to any direct (e.g., a direct wired connection between two devices or components) or indirect (e.g., wired and / or wireless connections between any number of devices or components connecting the operatively connected devices) connection. Thus, any path over which information can be transmitted can be considered an operative link.

[0102] While the examples discussed herein have been described with respect to a limited number of examples, those skilled in the art, having the benefit of this disclosure, will recognize that other examples may be developed that do not depart from the scope of the examples disclosed herein. Accordingly, the scope of the examples described herein should be limited only by the appended claims.

Claims

[1] A system comprising: one or more processors; and one or more non-transitory computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to: receiving, at a framework daemon of a workload execution framework executing on a first node of a set of nodes assigned by a workload manager to execute a workload, an indication to begin execution of the workload, the workload being divided into a plurality of execution units; accessing an execution context comprising the plurality of execution units, through the framework daemon and in response to the indication; claiming a first execution unit of the plurality of execution units for execution by the first node through the framework daemon by associating a first identifier of the first node with the first execution unit; to execute the first execution unit by the first node; by the framework daemon and after the first node has completed execution of the first execution unit, determine whether the execution context includes a second execution unit of the plurality of execution units that is unclaimed by accessing the execution context; by the framework daemon, and if the determination is that the execution context includes the unclaimed second execution unit, to claim the second execution unit for execution by the first node by associating the first identifier of the first node with the second execution unit; and the second execution unit is to be executed by the first node. [2] The system of claim 1, wherein execution of the instructions further causes the one or more processors to: detect, through a framework monitor of the workload execution framework, that a second node of the set of nodes executing the workload has failed; by the framework monitor and in response to determining that the second node has failed, send a second indication to corresponding framework daemons of non-failed nodes of the set of nodes that the second node has failed, the second indication comprising a second identifier of the second node; and successfully detaching the second node from each execution unit contained in the execution context by a framework daemon of the respective framework daemons and in response to the second indication. [3] The system of claim 2, wherein execution of the instructions further causes the one or more processors to: to send a scaling request comprising a replacement request to a workload manager by the one framework daemon of the respective framework daemons that has successfully detached the second node from an execution unit contained in the execution context. [4] The system of claim 1, wherein execution of the workload is complete when each node of the set of nodes has entered a lock state and each execution unit of the plurality of execution units has been executed by the set of nodes. [5] The system of claim 1, wherein the workload execution framework further comprises a framework monitor and the execution of the instructions further causes the one or more processors: monitor the execution of the workload on the group of nodes through the framework monitor; and adjust a number of nodes included in the set of nodes based on the monitoring by sending a scaling request to a workload manager. [6] The system of claim 5, wherein the scaling request comprises an increase request. [7] The system of claim 5, wherein the scaling request comprises a distance request. [8] A computer-implemented method comprising: Receiving, at a framework daemon of a workload execution framework executing on a first node of a set of nodes assigned by a workload manager to execute a workload, an indication to begin execution of the workload, wherein the workload is divided into a plurality of execution units; Accessing, by the framework daemon and in response to the indication, an execution context comprising the plurality of execution units; Requesting, by the framework daemon, a first execution unit from the plurality of execution units for execution by the first node by associating a first identifier of the first node with the first execution unit; Executing the first execution unit by the first node; Determining, by the framework daemon and after the first node has completed execution of the first execution unit, whether the execution context includes a second execution unit of the plurality of execution units that is unclaimed by accessing the execution context; Requesting the second execution unit for execution by the first node by associating, by the framework daemon, the first identifier of the first node with the second execution unit if the execution context determines that the execution context includes the unrequested second execution unit; and Execution of the second execution unit by the first node. [9] A computer-implemented method according to claim 8, further comprising: Determining, by a framework monitor of the workload execution framework, that a second node of the set of nodes executing the workload has failed; Sending, by the framework monitor and in response to the determination that the second node has failed, a second indication to corresponding framework daemons of non-failed nodes of the set of nodes that the second node has failed, the second indication comprising a second identifier of the second node; and successfully detaching the second node from each execution unit contained in the execution context by a framework daemon of the respective framework daemons and in response to the second indication. [10] The computer-implemented method of claim 9, further comprising: Sending a scaling request that includes a replacement request to a workload manager by one of the respective framework daemons that has successfully detached the second node from an execution unit contained in the execution context. [11] The computer-implemented method of claim 8, wherein execution of the workload is complete when each node of the node set has entered a lock state and each execution unit of the plurality of execution units has been executed by the node set. [12] The computer-implemented method of claim 8, wherein the workload execution framework further comprises a framework monitor, and the computer-implemented method further comprises: Monitoring the execution of the workload on the group of nodes through the framework monitor; and Adjust, based on monitoring, a number of nodes included in the set of nodes by sending a scaling request to a workload manager. [13] The computer-implemented method of claim 12, wherein the scaling request comprises an increase request. [14] The computer-implemented method of claim 12, wherein the scaling request comprises a distance request. [15] A non-transitory, computer-readable medium storing programming for execution by one or more processors, the programming comprising instructions to: Receiving, at a framework daemon of a workload execution framework executing on a first node of a set of nodes assigned by a workload manager to execute a workload, an indication to begin execution of the workload, wherein the workload is divided into a plurality of execution units; Accessing, by the framework daemon and in response to the indication, an execution context comprising the plurality of execution units; Claiming, by the framework daemon, a first execution unit of the plurality of execution units for execution by the first node by associating a first identifier of the first node with the first execution unit; Executing the first execution unit by the first node; Determining, by the framework daemon and after the first node has completed execution of the first execution unit, whether the execution context includes a second execution unit of the plurality of execution units that is unclaimed by accessing the execution context; Requesting, by the framework daemon and if the determination is that the execution context includes the unclaimed second execution unit, the second execution unit for execution by the first node by associating the first identifier of the first node with the second execution unit; and Execution of the second execution unit by the first node. [16] The non-transitory computer-readable medium of claim 15, wherein the programming comprises further instructions for: Determining, by a framework monitor of the workload execution framework, that a second node of the node set executing the workload has failed; Sending, by the framework monitor and in response to the determination that the second node has failed, a second indication to corresponding framework daemons of non-failed nodes of the set of nodes that the second node has failed, the second indication comprising a second identifier of the second node; and successfully detaching the second node from each execution unit contained in the execution context by a framework daemon of the respective framework daemons and in response to the second indication. [17] The non-transitory computer-readable medium of claim 16, wherein the programming comprises further instructions for: Sending a scaling request that includes a replacement request to a workload manager by one of the respective framework daemons that has successfully detached the second node from an execution unit contained in the execution context. [18] The non-transitory computer-readable medium of claim 15, wherein execution of the workload is complete when each node of the set of nodes has entered a lock state and each execution unit of the plurality of execution units has been executed by the set of nodes. [19] The non-transitory computer-readable medium of claim 15, wherein the programming comprises further instructions for: Monitoring the execution of the workload on the set of nodes through a framework monitor; and Adjust, based on monitoring, a number of nodes included in the set of nodes by sending a scaling request to a workload manager. [20] The non-transitory computer-readable medium of claim 19, wherein the scaling requirement comprises at least one of the following elements: a request for an increase; and a removal request.