Dynamic allocation of computing resources

Abstract

A method for performing distributed computations on a computing system using dynamically allocated computational resources using a computational resource manager includes storing information specifying an amount of computational resources associated with each of several program portions of a program, the program portions performing successive transformations of data, each program portion using computational resources granted by the computational resource manager to enable computations associated with that program portion to be performed in the computing system; requesting from the computational resource manager a first amount of computational resources associated with a first of the several program portions; receiving from the computational resource manager a second amount of computational resources that is less than the requested first amount of computational resources; performing computations associated with the first portion of the program using the second amount of computational resources; receiving from the computational resource manager an additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources; and performing additional computations associated with the first portion of the program using the additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 196,757, filed June 4, 2021, the entire contents of which are incorporated herein by reference.

[0002] This description relates to dynamic allocation of resources in a computing system. [Background technology]

[0003] Applications running on a computing system require a portion of the computing system's computational resources in order to run. Accordingly, the computing system must manage the allocation of its resources to applications running on the computing system. Examples of some of the resources allocated to an application include access to a portion of the computing system's memory, access to file data, and access to required processing power.

[0004] In a distributed computing system, computational resources (including data storage and processing resources) are distributed across multiple servers in one or more clusters that cooperate to execute data processing applications. In some examples, distributed computing systems use a centralized resource manager that schedules the execution of applications on the computing system and manages the allocation of the computing system's distributed resources to the applications. Examples of resource managers include "Hadoop YARN" and "Kubernetes." Summary of the Invention [Means for solving the problem]

[0005] In a general aspect, a method for performing distributed computations on a computing system using dynamically allocated computational resources using a computational resource manager includes storing information specifying an amount of computational resources associated with each of several program portions of a program, the program portions performing successive transformations of data, each program portion using computational resources granted by the computational resource manager to enable computations associated with that program portion to be performed on the computing system; requesting from the computational resource manager a first amount of computational resources associated with a first of the several program portions; receiving from the computational resource manager a second amount of computational resources that is less than the requested first amount of computational resources; performing computations associated with the first portion of the program using the second amount of computational resources; receiving from the computational resource manager an additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources; and performing additional computations associated with the first portion of the program using the additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources.

[0006] Aspects may include one or more of the following features.

[0007] The information specifying the amount of computational resources associated with each program portion of the several program portions of the program may include characteristics of one or more program components associated with each program portion of the several program portions. The characteristics may include a degree of parallelism associated with each of the one or more program components and an amount of computational resources required to perform computations associated with each of the one or more program components. The amount of computational resources associated with a program portion of the several program portions may be determined based at least in part on the degree of parallelism associated with each of the one or more program components and the amount of computational resources required to perform computations associated with each of the one or more program components.

[0008] Performing computations associated with the first portion of the program using the second amount of computational resources may include partitioning the first program portion into a number of sub-portions in response to the received second amount of computational resources, and performing computations associated with the first portion of the program may include executing a first sub-portion of the number of sub-portions while one or more other sub-portions of the number of sub-portions wait to perform computations.

[0009] Partitions of a program portion may be executed in a manner that preserves the order of execution of at least some of the program components. Partitions of a program portion may be executed in a manner that maximizes the use of received computational resources while preserving the order of execution of at least some of the program components. Maximizing the use of received computational resources may include using the received computational resources to perform computations for some, but not all, required instances of program components in a first portion of the program portion and associating the remaining instances of the program components with another portion of the program portion for later execution.

[0010] The step of performing computations associated with the first portion of the program using the second amount of computational resources may include performing computations for a first sub-portion of the plurality of sub-portions using the second amount of computational resources while one or more other sub-portions of the plurality of sub-portions wait to perform computations, The first sub-portion may be a sub-portion of the plurality of sub-portions configured to use most or all of the second amount of computational resources for execution.

[0011] The step of performing computations related to the first portion of the program using the additional amount of computational resources may include repartitioning the first program portion into updated sub-portions in response to the received second amount of computational resources and the received additional amount of computational resources.

[0012] The step of performing calculations associated with the first portion of the program using the additional amount of computational resources may include performing calculations associated with a first sub-portion of the updated plurality of sub-portions using the additional amount of computational resources while one or more other sub-portions of the updated plurality of sub-portions are waiting to perform calculations.

[0013] The first sub-portion of the updated plurality of sub-portions may be the sub-portion of the updated plurality of sub-portions that is configured to use most or all of the additional amount of computational resources to perform the computation.

[0014] Partitioning the first program portion into subportions may include partitioning the first program portion according to one or more characteristics of program components associated with the first program portion. The characteristics of the program portions may include a degree of parallelism associated with each of the one or more program components and an amount of computational resources required to perform computations associated with each of the one or more program components. A first one or more instances of the first program component may be partitioned into a first subportion, and a second one or more instances of the first program component may be partitioned into a second subportion.

[0015] The method may include relinquishing the second amount of computational resources and the additional amount of computational resources upon completion of computations associated with the first program portion. The method may include retaining at least a portion of the second amount of computational resources and the additional amount of computational resources upon completion of computations associated with the first program portion. The method may include using at least a portion of the retained computational resources to perform computations associated with a second program portion.

[0016] A third amount of the received computational resources may become unavailable during computation associated with the first program portion, and the method may further include requesting the third amount of the computational resources from a computational resource manager, receiving the third amount of the computational resources from the computational resource manager, and continuing to perform computation associated with the first program portion using the received third amount of the computational resources.

[0017] The computational resource manager may be opaque regarding the amount of computational resources available for the computing system. The method may include performing a computation associated with a second program portion of the number of program portions, including storing output data from a first program portion and reading and processing the stored output data. The method may include performing a computation associated with a second program portion of the number of program portions, including receiving and processing a stream of output data from the first program portion.

[0018] In another general aspect, a system for performing distributed computations using dynamically allocated computing system computational resources using a computational resource manager includes: a storage device for storing information specifying an amount of computational resources associated with each of several program portions of a program, the program portions performing successive transformations of data, each program portion using computational resources granted by the computational resource manager to enable that program portion to execute in the computing system; and at least one processor configured to request from the computational resource manager a first amount of computational resources associated with a first of the several program portions; receive from the computational resource manager a second amount of computational resources that is less than the requested first amount of computational resources; perform computations associated with the first portion of the program using the second amount of computational resources; receive from the computational resource manager an additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources; and perform computations associated with the first portion of the program using the additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources.

[0019] In another general aspect, a system for performing distributed computations using dynamically allocated computing system computational resources using a computational resource manager includes: means for storing information specifying an amount of computational resources associated with each of several program portions of a program, the program portions performing successive transformations of data, each program portion using computational resources granted by the computational resource manager to enable execution of computations associated with that program portion in the computing system; and means for processing configured to request from the computational resource manager a first amount of computational resources associated with a first of the several program portions, receive from the computational resource manager a second amount of computational resources that is less than the requested first amount of computational resources, perform computations associated with the first portion of the program using the second amount of computational resources, receive an additional amount of computational resources from the computational resource manager while performing computations associated with the first portion using the second amount of computational resources, and perform computations associated with the first portion of the program using the additional amount of computational resources while performing computations associated with the first portion using the second amount of computational resources.

[0020] In another general aspect, software stored in non-transitory form on a computer-readable medium for performing distributed computations using dynamically allocated computing system computational resources using a computational resource manager includes instructions to cause the computing system to store information specifying an amount of computational resources associated with each of several program portions of a program, the program portions performing successive transformations of data, each program portion using computational resources granted by the computational resource manager to enable performance of computations related to that program portion on the computing system; requesting from the computational resource manager a first amount of computational resources associated with a first of the several program portions; receiving from the computational resource manager a second amount of computational resources that is less than the requested first amount of computational resources; performing computations related to the first portion of the program using the second amount of computational resources; receiving from the computational resource manager an additional amount of computational resources while performing computations related to the first portion using the second amount of computational resources; and performing computations related to the first portion of the program using the additional amount of computational resources while performing computations related to the first portion using the second amount of computational resources.

[0021] Performing computations associated with a program or program portion is also referred to as executing the program or program portion.

[0022] A program can be defined as a dataflow graph, and program portions can be defined as components of the dataflow graph.

[0023] A program can be defined as a procedural program specification, and portions of the program can be defined as subroutines.

[0024] Aspects may include one or more of the following advantages.

[0025] Among other advantages, aspects dynamically allocate computational resources to portions (e.g., components) of a computer program (e.g., a data processing graph) in a resource-constrained computing environment where the amount of resources available to the program portions changes over time. Portions (sometimes referred to as "phases") of the program are enabled to partially execute a computation with fewer than all of their required computational resources and are enabled to incorporate additional computational resources while executing the computation as they become available, such as to complete the computation. Advantageously, program portions are less likely to be held up while waiting to be granted all their required resources. Advantageously, program portions can recover from failures of resources (e.g., nodes) in the computing system by dynamically allocating new resources to replace the failed resources.

[0026] Other features and advantages of the invention will become apparent from the following description and from the claims. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a block diagram of a system for dynamic allocation of distributed computing resources. [Figure 2a] FIG. 1 is a detailed block diagram of a system for dynamic allocation of distributed computing resources. [Figure 2b] FIG. 10 is a diagram illustrating an example of a data processing graph. [Figure 3] 1 is a flowchart illustrating a process for dynamically allocating computing resources. [Figure 4] FIG. 2c illustrates a resource request module requesting and receiving computational resources for the first phase of the data processing graph of FIG. 2b. [Figure 5]FIG. 2c illustrates the first phase of the data processing graph of FIG. 2b being executed using the computational resources of the execution module. [Figure 6] FIG. 2c shows the data processing graph of FIG. 2b at the completion of execution of the first phase of the graph; [Figure 7] FIG. 2c illustrates a resource request module requesting computational resources for a second phase of the data processing graph of FIG. 2b and receiving less computational resources than requested. [Figure 8] FIG. 2c shows how the second phase of the data processing graph of FIG. 2b is partitioned. [Figure 9] FIG. 10 illustrates additional computational resources becoming available while part of the second phase is being executed. [Figure 10] FIG. 2c illustrates the second phase of the data processing graph of FIG. 2b being repartitioned based on the availability of additional computational resources. [Figure 11] FIG. 2c shows the data processing graph of FIG. 2b at the completion of the first part of the second phase. [Figure 12] FIG. 2c illustrates a second portion of a second phase of the data processing graph of FIG. 2b being executed on previously received computational resources. [Figure 13] FIG. 2c shows the data processing graph of FIG. 2b at the completion of a second portion of a second phase. [Figure 14] FIG. 2B illustrates the first portion of the third phase of the data processing graph of FIG. 2b being executed on previously received computational resources, with additional computational resources being requested and received. [Figure 15] FIG. 2c shows the second part of the third phase of the data processing graph of FIG. 2b being executed. [Figure 16] FIG. 10 illustrates the failure of a second host of an execution module. [Figure 17]FIG. 2c shows the computational resources for the second part of the third phase of the data processing graph of FIG. 2b being requested and received on the execution module's new host. [Figure 18] FIG. 2c shows the second part of the third phase of the data processing graph of FIG. 2b being executed. [Figure 19] FIG. 2c shows the third phase of the data processing graph of FIG. 2b completing execution. DETAILED DESCRIPTION OF THE INVENTION

[0028] 1. System Overview 1 illustrates an example of a data processing system 100 in which the resource allocation techniques may be used. System 100 includes data sources 102, which may include one or more sources of data, such as storage devices or connections to online data streams, each of which may store or provide data in any of a variety of formats (e.g., database tables, spreadsheet files, flat text files, or native formats used by mainframes). Execution environment 104 includes execution module 112, resource manager 120, and resource request module 122. Execution environment 104 may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as, for example, a version of the UNIX operating system. Also, for example, execution environment 104 may include a multi-node parallel computing environment, which includes a configuration of computer systems using multiple central processing units (CPUs) or processor cores, either local (e.g., a multiprocessor system such as a symmetric multi-processing (SMP) computer), locally distributed (e.g., multiple processors connected as a cluster or massively parallel processing (MPP) system), remote or remotely distributed (e.g., multiple processors connected via a local area network (LAN) and / or wide-area network (WAN)), or any combination thereof.

[0029] The data storage system 116 is accessible by the execution environment 104 and the development environment 118. The development environment 118 is a system for developing programs that can be configured in various ways so that different interrelated program portions are associated with different target amounts of computing resources allocated for use at runtime. In some embodiments, these programs are data processing programs that process data during runtime, such as data received from the data source 102. One example of a data processing program is a data processing graph that includes vertices (representing data processing components or data sets) connected by directed links (representing work elements, i.e., the flow of data) between the vertices. Other forms of data processing programs are possible in accordance with the present invention. In addition to these data flow connections, some data processing graphs also have control flow connections to determine the flow of control between components. In such data processing graphs, program portions are components that are interrelated by data flow links. In other examples, program portions are submodules or other entities within a program that are separately endowed with computing resources for execution. Program portions are considered interrelated to the extent that the ability of the entire program to which they belong to execute depends on the ability of the individual program portions. Such interrelated or interdependent program portions may depend on each other for execution (e.g., one program portion may receive data from or provide data to another program portion), or program portions may be granted computing resources separately but overlap or depend on each other in various other ways (e.g., competing for a limited supply of computing resources).

[0030] For example, such an environment for developing graph-based computations is described in more detail in U.S. Patent Application Publication No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference. A system for performing such graph-based computations is described in U.S. Patent Application Publication No. 5,966,072, entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," which is incorporated herein by reference. The data processing graph generated by this system provides methods for moving information between processes and for moving information into and out of individual processes represented by components of the graph to define the order of execution for the processes. The system includes algorithms for selecting methods of inter-process communication from any available methods (e.g., communication paths following the links in the graph may use TCP / IP or UNIX domain sockets, or may use shared memory to pass data between processes).

[0031] The execution module 112 processes data from the data source 102 according to one or more data processing graphs 114 using computational resources allocated by the resource manager 120 to generate output data that is stored back at the data source 102, stored in a data storage system 116, or otherwise used. The storage device providing the data source 102 may be local to the execution environment 104, e.g., housed in a storage medium (e.g., a hard drive 108) connected to the computer hosting the execution environment 104, or remote from the execution environment 104, e.g., hosted on a remote system (e.g., a mainframe 110) that communicates with the computer hosting the execution environment 104 via a remote connection (e.g., provided by a cloud computing infrastructure). In some examples, the data source 102 comprises different forms of database systems that contain data that may be organized as records having values ​​for respective fields (also called "attributes" or "columns"), possibly including null values.

[0032] Resource manager 120 schedules the execution of one or more computer programs, such as data processing graph 114, on execution environment 104 and manages the allocation of the execution environment's resources to the data processing graph. As described in more detail below, for computer programs that include interrelated program portions, such as a data processing graph that includes multiple interdependent components, resource request module 122 interacts with resource manager 120 to dynamically allocate computing resources based on the availability of computing resources associated with execution module 112, which may change over time.

[0033] Referring to FIG. 2A, an alternative representation of data processing system 100 includes an execution environment 104, a data source 102, and a data processing program 224 (e.g., a data processing graph) for processing data from data source 102 using the computational resources of execution environment 104.

[0034] The execution environment 104 includes a resource request module 122, a resource manager 120, and an execution module 112. Among other features, the execution module 112 includes computational resources that may be distributed across multiple hosts (e.g., a computing cluster of servers). In FIG. 2A , there are three hosts: a first host H1 236, a second host H2 238, and a third host H3 240. Each host includes a finite amount of computational resources that collectively comprise the total computational resources of the execution module 112. Examples of computational resources managed and allocated by the resource manager 120 may include any of the following: a usage share of the host's processor (e.g., specified as virtual cores mapped to one or more physical cores of a multi-core processor); a portion of the host's volatile memory (e.g., specified as an amount of the host's main memory space); a portion of the host's non-volatile memory (e.g., specified as an amount of storage space on the host's hard disk drive); or a usage share of a communication channel (e.g., a portion of the bandwidth of the host's Ethernet interface). A single unit of computing resource may include multiple types of resources, such as a specified number of CPUs or virtual cores and a specified amount of main memory.

[0035] In FIG. 2A , for ease of explanation of the computational resource allocation methods described herein, the computational resources of a host are represented as computational resource units (shown as squares within the host), and those computational resource units are shown as all having the same granularity (i.e., the smallest size that can be granted). However, it is noted that the computational resources are not necessarily segmented into fixed units of equal granularity, but instead may be segmented into units of varying granularity or divided using other alternative techniques. Furthermore, for ease of explanation of the computational resource allocation methods described herein, all of the hosts in the execution module 112 of FIG. 2A are shown as having the same number (i.e., 9) of computational resource units. However, it is noted that different hosts may generally have different amounts of computational resources.

[0036] Resource manager 120 receives requests for computational resources and either grants or denies the requests based on the amount of available computational resources within the host of execution module 112. One example of such a resource manager 120 includes a Hadoop YARN resource manager, which can receive requests for computational resources to execute a computer program (or program portion) and, if sufficient computational resources are available, grants a “container” with some number of units of computational resources for use by the program. A container may be implemented as any suitable data structure containing a specific amount of computational resources, any information identifying a specific amount of computational resources, or any combination thereof. The computer program may then execute using the computational resources within the granted container. In some examples, a computer program may request multiple containers of resources from resource manager 120 at one time (e.g., several containers for running simultaneous instances of portions of the program). If sufficient resources are available for resource manager 120 to grant all of the requested containers to the computer program, resource manager 120 grants all of the requested containers to the computer program. Otherwise, based on available resources, resource manager 120 may grant only a portion of the requested containers (i.e., an integer number of containers less than the total number of requested containers), or resource manager 120 may grant none of the requested containers. In some embodiments, all of the computational resources associated with a given container come from a single host. Alternatively, in other embodiments, the resources for a given container may come from multiple hosts.

[0037] As described in more detail below, the resource request module 122 interacts with the resource manager 120 in a manner that enables dynamic allocation (e.g., incremental allocation, deallocation, or reallocation) of resources for the data processing program 224 as resource availability at the execution module 112 changes over time.

[0038] The data processing program 224 is a computer program specification for processing data received from the data source 102. Figure 2B is an example of a data processing graph-based specification of the data processing program 224, which includes several interrelated components, including a first component A 226, a second component B 228, a third component C 230, and a fourth component D 232.

[0039] In some examples, the execution of a data processing program (e.g., a data processing graph) is divided into multiple consecutive computational phases (sometimes called "program portions"), and each program component of a computer program (e.g., a node or component of a graph, or a subroutine of a procedural program) belongs to one of the computational phases. Generally, all program components belonging to a computational phase must complete their processing before program components belonging to the next subsequent computational phase can begin their processing. For example, the data processing program 224 of FIG. 2 is a data processing graph that includes three computational phases: a first computational phase 227, a second computational phase 229, and a third computational phase 231. A first component A 226 belongs to the first computational phase 227, a second component B 228 and a third component C 230 both belong to the second computational phase 229, and a fourth component D 232 belongs to the third computational phase 231.

[0040] Additionally, each program component of a computer program may be associated with a computational resource budget that specifies the amount of resources required for the program component on the execution module 112, and a “layout” constraint that specifies the degree of parallelism of the program component. A shorthand notation for the computational resource budget and layout of a component is “AxB,” where A is the computational resource budget and B is the layout constraint. In the example data processing graph of FIG. 2B , a first component A 226 is annotated as “4x2” because it has a computational resource budget of “4” resource units and a layout constraint of “2” that specifies that two instances (or plies) of the component are to be executed in parallel. A second component B 228 is annotated as “5x3” because it has a computational resource budget of “5” resource units and a layout constraint of “3” that specifies that three instances of the component are to be executed in parallel. The third component C 230 is annotated as "4x1" because it has a computational resource budget of "4" resource units and a layout constraint of "1" specifying that only one instance of the component is to be executed. The fourth component B 232 is annotated as "1x6" because it has a computational resource budget of "1" resource unit and a layout constraint of "6" specifying that six instances of the component are to be executed in parallel.

[0041] Referring again to FIG. 2A , each of the program components of the data processing program 224 is configured to process data elements (e.g., blocks, files, or records) from a dataset of the data source 102. In FIG. 2A , a first program component A 226 processes a first data element 252 from a first dataset 242, a second program component B 228 processes a second data element 254 from a second dataset 244, a third program component C 230 processes a third data element 256 from a third dataset 246, and a fourth program component D 232 processes a fourth data element 258 from a fourth dataset 248. In some examples, a block is a portion of data in a file that includes one or more records. In the examples described herein, each block of data elements in a dataset is processed by a corresponding instance of a program component associated with the dataset. However, it is noted that other allocations of data to instances of program components are possible. For example, a dataset associated with each program component may be analyzed to determine a target number of instances of different program components, and thus a target amount of computing resources, based on the number of data elements. In other examples, the target amount of computing resources may not necessarily be based on the amount of data in the associated dataset, but may instead (or additionally) depend on other factors, such as historical or estimated performance data. It is noted that there is no requirement that each program component be associated with a different dataset. For example, several program components may all process data from one or more of the same datasets.

[0042] 2 Dynamic resource allocation In resource-abundant scenarios, the resource request module 122 can allocate resources to phases of the data processing program 224 without issue. Each phase uses its allocated resources to produce results that may be used by subsequent phases of the data processing program. When a phase is completed, its allocated computational resources are relinquished. However, in some examples, the execution module 112 has limited computational resources, which may result in the resource request module 122 receiving fewer than all of the computational resources it requests for a phase. In such examples, rather than waiting to execute a phase until the remainder of the resources for the phase are available, aspects described herein use a dynamic resource allocation process that executes part of the phase with already allocated resources and executes further parts of the phase as additional computational resources become available.

[0043] Referring to FIG. 3 , a resource allocation process 300 dynamically allocates computational resources for executing a data processing program 224 when the execution module 112 has limited computational resources. The process 300 receives representations of multiple phases 360, each of which includes information related to program components within the phase (e.g., the program component's computational resource requirements and degree of parallelism). To facilitate description of the process 300, each representation of a phase 360 ​​is considered to begin with a single sub-phase that includes all program components of the phase. For each representation of a phase 360, and for each sub-phase of the phase 360, a first step 362 of the process 300 requests all computational resources needed for the sub-phase to execute. When all requested computational resources for a sub-phase are received (alternatively referred to as "allocated" or "granted") in a second step 364 of the process 300, a third step 365 executes the sub-phase using the received computational resources. Otherwise, if less than all of the requested computing resources are received, a fourth step 366 of process 300 partitions the sub-phase into two (or in some instances multiple) sub-phases: a first sub-phase and a second sub-phase.

[0044] For each sub-phase, a fifth step 368 of process 300 executes the first sub-phase using the received computational resources, while the second sub-phase waits for the resources to become available. If additional resources become available during execution of a sub-phase in a sixth step 370 of process 300, a seventh step 372 of process 300 extends the first sub-phase (and contracts the second sub-phase) so that it can use the additional received computational resources (e.g., by adding instances of components from the second sub-phase to the first sub-phase). In an eighth step 374 of process 300, the extended first sub-phase is executed.

[0045] As each sub-phase (or extended sub-phase) completes execution, the results of the execution are stored (e.g., in memory or on disk). Process 300 iterates through each sub-phase (e.g., an added second sub-phase), with each sub-phase reading any results stored by the previous sub-phase and executing using the allocated computational resources, as described above, until all sub-phases of the phase have been executed. This process repeats for each phase.

[0046] 3 Phase division 4, in an exemplary application of the process 300 of FIG. 3 to the data processing program 224 (i.e., the data processing graph) of FIG. 2B, a first step 362 of the process 300 begins with the resource request module 122 requesting all resources needed to execute a single sub-phase of the first phase 227 of the data processing program 224. In this case, the single sub-phase of the first phase 227 requires eight computational resource units because it includes two instances of the first program component A 226, which requires four computational resource units per instance to execute. The 4x2 request is sent to the resource manager 120, which responds to the resource request module 112 by granting the requested 4x2 computational resource units for the single sub-phase of the first phase 227. The granted computational resource units are shown in bold outline within the execution module 112, with four computational resource units granted on the first host H1 236 and four computational resource units granted on the third host H3 240.

[0047] 5 , the second step 364 of the process 300 determines that all requested computational resources have been received, and therefore the process 300 proceeds to the third step 365. The third step 365 uses the granted computational resource units to execute a single sub-phase of the first phase 227, which includes two instances of the first program component A 226, with the first instance (i.e., A1) of the first program component A 226 executing on the first host H1 236 and the second instance (i.e., A2) of the first program component A 226 executing on the third host H3 240. Note that in the example of FIG. 5 , the program component instances, when executed, are required to use only computational resource units belonging to a single host. However, in other examples, the program component instances may execute using computational resource units belonging to multiple hosts.

[0048] 6, when the two instances of the first program component A 226 that use the received computational resource units have completed execution and the results of the execution have been stored, the resource request module 122 sends a "Done" message to the resource manager 120 indicating that the execution is complete. The resource request module 122 marks the instances of the first program component A 226 as completed (shown as a check mark in the figure). With the two instances of the first program component A 226 having finished executing, the single sub-phase of the first phase 227 is complete and the granted computational resources are relinquished.

[0049] 7 , the process 300 proceeds to the second phase 229 of the data processing graph 224. The first step 362 of the process 300 begins with the resource request module 122 requesting all of the computational resources needed to execute a single sub-phase of the second phase 229. In this case, the single sub-phase of the second phase 229 requires 19 computational resource units because it includes three instances of the second program component B 228, each of which requires five computational resource units per instance to execute, and one instance of the third program component C 230, each of which requires four computational resource units per instance to execute. The 5x3 and 4x1 computational resource requests are sent to the resource manager 120, which responds by granting 5x2 computational resource units to the resource request module 112. The granted computational resource units are shown in bold outline within the execution module 112, with five computational resource units granted on the first host H1 236 and five computational resource units granted on the third host H3 240.

[0050] 8 , a second step 364 of the process 300 determines that fewer than all requested computational resource units for a single sub-phase of the second phase 229 have been granted. Accordingly, the process 300 proceeds to a fourth step 366, which partitions the single sub-phase of the second phase 229 into two sub-phases, a first sub-phase 229a and a second sub-phase 229b. Generally, the partitioning of the sub-phases takes into account the order of execution of the program components of the data processing program. For example, the second program component B 228 of the data processing program 224 is designated to execute before the third program component C 230, and therefore, the partitioning of the second phase 229 is executed in a manner that ensures that all instances of the second program component B 228 are allocated resources before the third program component C 238 is allocated resources. 8, only "5x2" computational resource units are granted, and therefore only two instances of the second program component B 228 are included in the first sub-phase 229a. There are insufficient granted computational resources to run the remaining instances of the second program component B 228 and the third program component C 228 in the second phase 229, and therefore those instances are included in the second sub-phase 229b.

[0051] That is, the first sub-phase 229a includes a first instance (i.e., B1) of the second program component B 228 and a second instance (i.e., B2) of the second program component B 228. The first and second instances of the second program component 228 each require five computational resource units to execute and therefore may execute using the five computational resource units granted on the first host H1 236 and the five computational resource units granted on the third host H3 240. In a fifth step 368 of the process 300, the first and second instances (B1, B2) of the second program component 228 begin execution using the five computational resource units granted on the first host H1 236 and the five computational resource units granted on the third host H3 240.

[0052] 9 , in a sixth step 370 of the process 300, while the first sub-phase 229a is being executed, the resource manager 120 sends a message to the resource request module 122 indicating that another “five” computational resource units have become available on the second host H2 238. The newly granted computational resource units are shown in bold outline in the execution module 112, with five computational resource units granted on the second host H2 238.

[0053] 10 , process 300 proceeds to a seventh step 372, which extends first sub-phase 229a to include a third instance (i.e., B3) of second program component 228, which can use the newly granted “five” computational resource units for execution. The system may attempt to complete all instances of program component (B) before including instances of other program components (C) to make the results of program component (B) available, or the system may always attempt to utilize all available resources for execution, resulting in B3 being selected instead of C in FIG. 10 . An eighth step 374 of the process begins execution of the extended first sub-phase 229a by executing the third instance (B3) of the second program component on the second host H2 238.

[0054] 11 , when the execution of the three instances of second program component B 228 using the received computational resource units is completed and the results of the execution are stored, resource request module 122 sends a “Done” message to resource manager 120 indicating that the execution is complete. Resource request module 122 marks the instances of second program component B 228 as completed (shown as a check mark in the figure). With the three instances of second program component B 228 having finished executing, first sub-phase 229a is complete.

[0055] In some examples, when execution of a phase or sub-phase is completed, all received computational resources are relinquished. In other examples, at least a portion of the granted computational resources are retained for execution of instances of program components of subsequent phases, preferably without requiring module 122 to send a resource request to manager 120. For example, in Figure 11, five computational resource units belonging to first host H1 236 are retained for execution of an instance of third program component C 230 of second sub-phase 229b, which requires four computational resource units to execute.

[0056] Referring to FIG. 12 , after completion of the first sub-phase 229a, the process 300 begins execution of the second sub-phase 229b by executing an instance of the third program component C 230 on the retained computational resource units from the first host H1 236. Resources do not need to be claimed by step 362, and the second sub-phase 229b is executed in step 365. Note that in the example of FIG. 12 , only four of the five retained computational resource units on the first host H1 236 are utilized. This is due to the fact that some systems grant computational resources as “blocks” or “containers” that are treated as a single, indivisible unit that must be retained or discarded as a whole. As a result, reusing computational resources may require balancing the benefits of reusing resources against the potential waste caused by reusing resources.

[0057] 13, when the execution of the instance of the third program component C 230 using the granted computational resource units is completed and the results of the execution are stored, the resource request module 122 sends a "Done" message back to the resource manager 120 indicating that the execution is complete. The resource request module 122 marks the instance of the third program component C 230 as completed (shown as a check mark in the figure). With the instance of the third program component C 230 finished executing, the second sub-phase 229b (and second phase 229) is complete. Again, the granted computational resources on the first host H1 236 are reserved for the execution of the program component instance in the next phase.

[0058] 14, after completion of the second phase 229, the process 300 begins execution of the third phase 231 by executing a first instance D1 of the fourth program component D 232 on the computational resource units maintained on the first host H1 236. Here, because D1 requires only one computational resource unit to execute, only one of the five maintained computational resource units on the first host H1 236 is utilized. Generally, only one instance of a program component may be executed in a resource container.

[0059] In a first step 362 of process 300, resource request module 122 requests the computational resources needed to execute a single sub-phase of third phase 331. In this case, the single sub-phase of third phase 331 requires six computational resource units because it includes six instances of fourth program component D 232, each of which requires one computational resource unit to execute. However, the first instance D1 of fourth program component D 232 is allocated to reserved computational resources on the first host H1 236, so a request for 1x5 computational resources is sent to resource manager 120 for the remaining five instances of fourth program component D 232. Resource manager 120 responds to resource request module 112 by granting 1x5 computational resource units. The granted computational resource units are shown in bold outline within execution module 112, with five computational resource units granted on second host H2 238.

[0060] 15, the second step 364 of the process 300 determines that all requested computational resources have been received for the single sub-phase, and therefore the process 300 proceeds to the third step 365. The third step 365 uses the granted computational resource units to initiate execution of the remaining five instances of the fourth program component D 232 of the single sub-phase of the third phase 231, and all five instances of the fourth program component (i.e., D2-D6) execute on the second host H2 238.

[0061] 4. Rollback In some examples, a host running an instance of a program component may experience a failure during execution. Referring to Figure 16, a second host H2 238 running five instances of a fourth program component (i.e., D2-D6) experiences a failure, while a first host H1 234 continues to run a first instance D1 of the fourth program component.

[0062] 17 , to recover from the failure of the second host H2 238, the process 300 rolls back the execution of all instances of the program components in the third phase 231 to the initial state of the phase (e.g., the state stored at the completion of the second phase 229). In some examples, the execution steps 365, 368 of the process 300 include detecting whether computational resources have been disabled or otherwise become unavailable and rolling back execution (not shown). After the execution is rolled back, the resource request module 122 sends a “1x5” computational resource request to the resource manager 120 for the five instances of the fourth program component D 232 that were executing on the second host H2 238. The resource manager 120 responds to the resource request module 112 by granting “1x5” computational resource units on the third host H3 240. The granted computational resource units are indicated by a bold outline within the execution module 112.

[0063] Referring to FIG. 18, five instances (D2-D6) of the fourth program component D 232 begin execution using the granted computational resource units on the third host H2 240.

[0064] 19, when the six instances of the fourth program component D 232 that use the granted computational resource units have completed execution and the results of the execution have been stored, the resource request module 122 sends a "Done" message to the resource manager 120 indicating that the execution is complete. The resource request module 122 marks the instances of the fourth program component D 232 as completed (shown as a check mark in the figure). With the six instances of the fourth program component D 232 having finished executing, the third and final phase 231 of the data processing graph is complete and the granted computational resources are relinquished.

[0065] 5 Alternative methods In some cases, the execution of phases of a data processing graph may overlap, and output data from one phase may be streamed to a subsequent phase rather than stored to disk.

[0066] In some examples, the resource manager does not provide any indication of the amount of computational resources available on the computing system. Rather, the resource manager accepts requests for computational resources and fulfills the requests (fully or partially) based on the computational resources available at the time of the request.

[0067] In the example above, execution is rolled back due to a node failure. In other examples, execution is rolled back, usually without warning, when resource manager 120 "disables" or "preempts" computational resources from resource request module 122. For example, if another application requests resources on node H2 238, resource manager 120 may determine that the other application is more important and disable the computational resources already granted to the application. The rollback procedure described above is used to allocate new computational resources to replace the disabled resources.

[0068] In some examples, such as the example above, after the computational resources for a phase are canceled, the entire phase is rolled back and restarted. In other examples, only the program components of the phase that used the previously avoided or disabled resources have their execution rolled back. Doing so advantageously avoids the redundant work of re-executing program components that did not have their computational resources disabled.

[0069] In the above example, the resource manager receives requests for computational resources and grants the resources if those resources are available. In other examples, rather than receiving requests, the computational resource manager offers computational resources to programs that wish to run in the execution environment. In such an arrangement, rather than making requests to the resource manager, programs listen to a stream of "offers" about available computational resources, and the programs choose which, if any, of the offers they want to accept. One example of a resource manager that works in this manner is Apache Mesos.

[0070] Performing a computation associated with a program or program portion may also be referred to as executing the program or program portion. A program may be specified as a data flow graph, and program portions may be specified as components of the data flow graph. A program may be specified as a procedural program specification, and program portions may be specified as subroutines.

[0071] Some partitions of the program may be executed in a manner that preserves the order of execution of at least some of the program components. Some partitions of the program may be executed in a manner that maximizes the use of received computational resources while preserving the order of execution of at least some of the program components.

[0072] 6. Embodiments The computational resource allocation techniques described above can be implemented, for example, using a programmable computing system executing suitable software instructions, or in suitable hardware such as a field-programmable gate array (FPGA), or in some hybrid form. For example, in a programmed manner, the software may include one or more computer program procedures executing on one or more programmed or programmable computing systems (which may be of various architectures, such as distributed, client / server, or grid), each including at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), and at least one user interface (for receiving input using at least one input device or port and for providing output using at least one output device or port). The software may include one or more modules of a larger program that provides services related to the design, configuration, and execution of a data processing graph, for example. The program modules (e.g., elements of a data processing graph) may be implemented as data structures or other organized data conforming to a data model stored in a data repository.

[0073] Software may be stored in non-transitory form, such as embodied in a volatile or non-volatile storage medium or any other non-transitory medium, using the physical characteristics of the medium (e.g., surface pits and lands, magnetic domains, or electrical charges) for a period of time (e.g., the time between refresh periods of a dynamic memory device such as dynamic RAM). In preparation for loading instructions, the software may be provided on a tangible non-transitory medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or may be distributed (e.g., encoded in a propagated signal) over a network communication medium to a tangible non-transitory medium of a computing system on which the software is executed. Some or all of the processing may be performed on a dedicated computer or using dedicated hardware such as a coprocessor or field programmable gate array (FPGA) or dedicated application-specific integrated circuit (ASIC). Processing may be performed in a distributed manner, where different parts of the computation specified by the software are performed by different computing elements. Each such computer program is preferably stored in or downloaded to a computer-readable storage medium (e.g., solid-state memory or medium, or magnetic or optical medium) of a storage device that can be accessed by a general-purpose or special-purpose programmable computer, so as to configure and operate the computer when the medium of the storage device is read by the computer to perform the processes described herein. The system of the present invention may also be considered to be implemented as a tangible, non-transitory medium configured with a computer program, the medium so configured causing the computer to operate in a specific, predefined manner to perform one or more of the process steps described herein.

[0074] Several embodiments of the present invention have been described. However, it should be understood that the above description is intended to illustrate, and not to limit, the scope of the present invention, which is defined by the appended claims. Accordingly, other embodiments are within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. Furthermore, some of the steps described above may be order-independent and, therefore, may be performed in an order different from that described.

Claims

1. 1. A method for performing distributed computations on a computing system using computational resources dynamically allocated using a computational resource manager, comprising: storing information specifying an amount of computational resources associated with each of a plurality of program portions of a program, the program portions performing successive transformations of data, each program portion using computational resources granted by the computational resource manager to enable the program portion to execute on the computing system; requesting from the computational resource manager a first amount of computational resources associated with a first program portion of the plurality of program portions; receiving, from the computational resource manager, a second amount of computational resources that is less than the requested first amount of computational resources when the first amount of computational resources is unavailable due to limited computational resources of an execution module of the computing system; performing computations associated with a first portion of the program using the second amount of computational resources; receiving, from the computational resource manager, an additional amount of computational resources when the additional amount of computational resources becomes available while performing computations related to the first portion of the program using the second amount of computational resources; performing additional calculations associated with the first portion of the program using the additional amount of computational resources while performing calculations associated with the first portion using the second amount of computational resources; The method comprising:

2. 2. The method of claim 1, wherein the information specifying the amount of computing resources associated with each of the plurality of program portions of the program includes characteristics of one or more program components associated with the each of the plurality of program portions.

3. 3. The method of claim 2, wherein the characteristics include a degree of parallelism associated with each of the one or more program components and an amount of computing resources required for each of the one or more program components to execute.

4. 4. The method of claim 3, wherein an amount of computational resources associated with a program portion of the plurality of program portions is determined based at least in part on a degree of parallelism associated with each of the one or more program components and an amount of the computational resources required to execute each of the one or more program components.

5. 5. The method of claim 1, wherein the step of performing computations associated with the first portion of the program using the second amount of computational resources comprises partitioning the first program portion into a plurality of sub-portions in response to the received second amount of computational resources.

6. 6. The method of claim 5, wherein performing computations associated with the first portion of the program using the second amount of computational resources comprises performing computations associated with a first sub-portion of the plurality of sub-portions using the second amount of computational resources while one or more other sub-portions of the plurality of sub-portions are waiting to perform computations, the first sub-portion being a sub-portion of the plurality of sub-portions that is configured to use most or all of the second amount of computational resources for execution.

7. 6. The method of claim 5, wherein performing computations associated with the first portion of the program using the additional amount of computational resources comprises repartitioning the first program portion into updated subportions in response to the received second amount of computational resources and the received additional amount of computational resources.

8. 8. The method of claim 7, wherein performing computations associated with the first portion of the program using the additional amount of computational resources comprises performing computations associated with a first sub-portion of the updated plurality of sub-portions using the additional amount of computational resources while one or more other sub-portions of the updated plurality of sub-portions wait to perform computations, the first sub-portion of the updated plurality of sub-portions being the sub-portion of the updated plurality of sub-portions configured to use most or all of the additional amount of computational resources to perform computations.

9. 8. The method of claim 7, wherein performing calculations associated with the first program portion of the program using the additional amount of computational resources comprises performing calculations associated with a first sub-portion of the plurality of sub-portions using the second amount of computational resources and performing calculations associated with a further portion of the first portion using the additional amount of computational resources.

10. 8. The method of claim 7, wherein a first sub-portion of the updated plurality of sub-portions is an extended version of the first sub-portion of the plurality of sub-portions formed by adding one or more instances of one or more program components of the first program portion from one or more other sub-portions of the plurality of sub-portions to the first sub-portion of the plurality of sub-portions to obtain the first sub-portion of the updated plurality of sub-portions.

11. 6. The method of claim 5, wherein partitioning the first program portion into a plurality of sub-portions includes partitioning the first program portion according to one or more characteristics of program components associated with the first program portion.

12. The characteristics of the plurality of program portions are 12. The method of claim 11, wherein the degree of parallelism associated with each of the one or more program components and the amount of computing resources required for each of the one or more program components to execute.

13. 13. The method of claim 12, wherein a first one or more instances of a first program component are partitioned into a first sub-portion and a second one or more instances of the first program component are partitioned into a second sub-portion.

14. The method of any of claims 1 to 4, further comprising the step of relinquishing the second amount of computing resources and the additional amount of computing resources upon completion of execution of the first program portion.

15. The method of any of claims 1 to 4, further comprising the step of retaining at least a portion of the second amount of computing resources and the additional amount of computing resources upon completion of execution of the first program portion.

16. 16. The method of claim 15, further comprising using at least a portion of the retained computational resources to perform computations associated with a second portion of the program.

17. 5. The method of claim 1, wherein a third amount of the received computational resource becomes unavailable during execution of computations related to the first program portion, requesting the third amount of computational resources from the computational resource manager; receiving the third amount of computational resources from the computational resource manager; and continuing to perform computations associated with the first program portion using the received third amount of computational resources.

18. The method of any of claims 1 to 4, wherein the computational resource manager does not provide any indication regarding the amount of computational resources available for the computing system.

19. 5. The method of claim 1, further comprising the step of performing a calculation associated with a second program portion of the plurality of program portions, the calculation including storing output data from the first program portion, and reading and processing the stored output data.

20. 5. The method of claim 1, further comprising the step of performing a computation associated with a second program portion of the plurality of program portions, the computation comprising receiving and processing a stream of output data from the first program portion.

21. A system for performing distributed computations using dynamically allocated computing system computational resources using a computational resource manager, the system comprising: a processor; and a storage medium storing instructions that, when executed by the processor, cause the system to perform a method according to any one of claims 1 to 4.

22. A non-transitory computer readable medium storing instructions for causing a computing system to perform the method according to any one of claims 1 to 4.

Images