Method, system and storage device for dynamic distribution of container images

Abstract

A method implemented by a data processing system, the method comprising: accessing a container image including a first application and a second application; determining, by the data processing system, a number of parallel executions for a given module of the first application; generating, for the given module, a plurality of instances of the container image according to the determined number of parallel executions, and configuring each instance to execute the given module of the first application; causing each of the plurality of configured instances to execute on one or more of the host systems; and, for at least one of the plurality of configured instances, causing, by the second application of the configured instance, communication between the data processing system and one or more of the host systems executing the configured instance.

Description

[0001] Priority claim

[0002] This application claims priority to U.S. patent application serial number 16 / 656,886, filed on October 18, 2019, which claims priority under 35 U.S.C. §119(e) to U.S. patent application serial number 62 / 844,430, filed on May 7, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0003] Applications running on a computing system require a portion of the computing system's computing resources to run. Therefore, the computing system must manage the allocation of resources to the applications running on it. Some examples of resources allocated to applications include access to a portion of the computing system's memory, access to file data, and access to a required amount of processing power. Summary of the Invention

[0004] In one aspect, a method implemented by a data processing system for causing instances of a container image to execute on multiple host systems is generally disclosed, wherein each container image includes a first application having multiple modules, and wherein the instances are configured to execute a given module based on a determined number of parallel executions of the given module, wherein the method includes: accessing a container image including the first application and a second application, wherein the second application causes communication between the data processing system and the host system executing the instance of the container image; determining, by the data processing system, the number of parallel executions of a given module of the first application; for the given module, generating, based on the determined number of parallel executions, multiple instances of the container image, wherein each instance includes the first application and the second application; for each instance, configuring the instance to execute a given module of the first application; causing each of the multiple configured instances to execute on one or more of the host systems; and, for at least one of the multiple configured instances, causing, by the second application of the configured instance, communication between the data processing system and one or more of the host systems executing the configured instance. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of these methods.

[0005] In this example, the operations include accessing a specification that specifies a number of parallel executions for a given module. The container image is a first container image, and wherein the method further includes: accessing an instance of a second container image from a hardware storage device, wherein the second container image includes the first application and specifies the amount of parallelism for each module in the second application; and storing the second container image on an execution system. The operations also include transmitting a request from the instance of the second container image to an interface included in the execution system to generate multiple instances of the container image according to the determined number of parallel executions. The operations also include assigning each of the multiple configured instances to one or more of the host systems. The assignment is performed dynamically by assigning each configured instance at runtime, and the assignments are not predetermined. The first application is a dataflow graph having multiple components, and the operations further include: for each component, generating multiple instances of the container image according to the amount of parallel execution determined for the component; and dynamically running the dataflow graph on multiple nodes to obtain a dynamic level of parallelism by: for each component, dynamically assigning the generated instances to one or more nodes of the host systems; and causing each of the multiple assigned instances to execute on the one or more nodes of the host systems. In some examples, the given module is a first module, the generated multiple instances are multiple first instances, and the method further includes: determining, by the data processing system, a number of parallel executions of a second module of the first application; for the second module, generating multiple second instances of the container image according to the number of parallel executions determined for the second module, wherein each second instance includes the first application and the second application; for each second instance, configuring the second instance to execute the second module of the first application; causing each of the multiple configured second instances to execute on one or more host systems among the host systems; and causing a communication channel to be established between one of the first instances and one of the second instances, wherein one of the first instances outputs data and transmits the output data to one of the second instances through the communication channel.

[0006] In some examples, communication between the data processing system and one or more of the host systems executing the configured instance includes: transmitting monitoring data by the given module to the second application, and passing the monitoring data by the second application to the data processing system, wherein the monitoring data is configured to be used by the data processing system to track when the given module has completed execution so that the data processing system can instruct another module to execute. In some examples, the given module is a first module, wherein the generated multiple instances are multiple first instances, and wherein the operations further include: determining, by the data processing system, a number of parallel executions of a second module of the first application; generating, for the second module, multiple second instances of the container image according to the number of parallel executions determined for the second module, wherein each second instance includes the first application and the second application; for each second instance, configuring the second instance to execute the second module of the first application; causing each of the multiple configured second instances to execute on one or more of the host systems; obtaining an address of one of the second instances from a local data storage area; and providing output from one of the first instances to one of the second instances from one of the first instances based on the obtained address.

[0007] Other embodiments of these aspects include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of these methods together or in any combination.

[0008] Among other advantages, aspects enable significant flexibility (in identifying resources for modules in a host container) and / or enable a reduction in computing resources when executing an application (despite launching the application from within a container). The techniques described herein facilitate more efficient and / or flexible use of computing resources of a computing system executing a computer program, and thereby enhance and ensure proper internal functionality of the computing system. Another technical effect of the techniques described herein is the effect on computer programs, such as data processing graphs, executed on a computing system. Further, a greater number of computer program parts (modules) may be able to be executed simultaneously, and / or some computer program parts may be able to start more quickly by not having to wait a long time to acquire the necessary computing resources of the computing system required for execution.

[0009] Other features and advantages of the invention will be apparent from the following description, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 、 Figure 3B and Figure 4A 、 Figure 4B Each is a block diagram of a system for dynamic distribution of mirror instances.

[0011] Figures 2A to 2F A time series showing the dynamic distribution of container images.

[0012] Figure 3A is a graphic representation of a data processing graph.

[0013] Figure 5 is a flow chart of a process for dynamic distribution of container images. DETAILED DESCRIPTION

[0014] Data processing graphs are dynamically run on multiple nodes to achieve dynamic levels of parallelism. These data processing graphs are computationally intensive, with a large amount of computational disk input and output (I / O). To reduce disk I / O, these data processing graphs are launched from within containers that share a kernel with the host operating system (of the system hosting the container). Typically, a container includes an abstract data type (ADT) whose class, data structure, or instance is a collection of other objects. A container can be implemented as any suitable data structure for containing a specific amount of computing resources or containing any information identifying a specific amount of computing resources, or any combination thereof. The container stores objects in an organized manner that follows specific access rules. Scheduling, networking, disk I / O, and access to hardware are provided by the kernel. To take advantage of the advantages of containers, the system described herein dynamically distributes these containers (including data processing graphs) across multiple nodes. The distribution is dynamic because, before allocating resources to the nodes for executing the containers, it is unknown (e.g., not previously defined or specified) which nodes will execute which containers. By implementing this dynamic distribution of containers (including data processing graphs), the system achieves significant flexibility (in identifying resources for hosting containers) and achieves a reduction in computing resources when executing data processing graphs (despite launching the graphs from within containers).

[0015] Typically, a data processing graph (e.g., a computational graph) will include multiple components, each of which encapsulates executable logic. Each component can be specified to run in parallel in a certain number of ways, for example, across multiple different machines. Thus, the first component can be executed on five different machines, and the second component can be executed on two different machines. In this example, the first component outputs data to the second component. In other words, the first component is upstream of the second component. One advantage of the system described herein is that the system dynamically generates connections (e.g., communication channels or pipes) between the first component (included in a container executing on one machine) and the second component (included in another container executing on a different machine), for example, at runtime. The connections between components do not need to be pre-specified or pre-configured. Using the techniques described herein, connections can be dynamically established at runtime and on the system hosting the containers. In contrast, prior art systems cannot dynamically generate these connections. Therefore, in prior art systems, all upstream applications must complete execution before data is transmitted to downstream applications, resulting in delays in the execution of the applications. In other words, in prior art systems, applications run serially, rather than continuously as described herein. Specifically, using the system described herein, dataflow graphs (and their attendant components) execute continuously and in real time with reduced latency because a first component does not need to wait for all first components to complete execution before transferring output data to a second component.

[0016] refer to Figure 1 , shows a network environment 100 in which instances of images are started in a dynamic distribution. Typically, an image includes a file that generates an application runtime environment when executed, which includes an application (e.g., an executable application) and a runtime environment for executing the application (e.g., system tools, libraries, system binaries, and settings). An instance of an image refers to a specific occurrence of an image. There are various types of application runtime environments, including, for example, containers. Typically, an application runtime environment uses lightweight operating system-level virtualization by sharing a kernel with the host operating system. Therefore, scheduling, networking, disk input / output, and access to hardware are provided by the kernel. Therefore, compared to a virtual machine, an application runtime environment has faster startup time, better resource distribution, direct hardware access, and less redundancy. Typically, dynamic distribution (also referred to herein as dynamic layout) refers to the amount of parallelism implemented when executing an application (or a portion thereof), and the allocation (i.e., dynamic allocation) of resources required to implement parallelism at runtime, as described in US2018 / 0165129 A1, the entire contents of which are incorporated herein by reference. For example, instances of container images (as described herein) are dynamically assigned to nodes on a host system, eg, by assignment at runtime and the assignment is not predetermined.

[0017] Environment 100 includes an execution system 106 and a data store 115. Execution system 106 is a system for automating application deployment, scaling, and management (e.g., Kubernetes). System). The data storage device 115 stores an application image 118 and an agent image 119. The agent image 119 is a container image - an image of a container. Typically, an image includes a copy of the contents of a storage device. In this example, the application image 118 is also a container image. Hereinafter, without limitation and for the purpose of convenience, the agent image 119 may be referred to as the container image 119. Typically, an application image includes an image of an application that is configured to request the launch of other images and further instruct those other images on how to execute. In this example, the application image is also a container image. The agent image 119 includes an image of a container that includes an application that is being executed and is configured to receive instructions on how to execute from the application image 118 (e.g., a data processing graph as described herein).

[0018] The execution system 106 includes an interface 111 for communicating with external systems (e.g., nodes). The execution system 106 also retrieves an application image 118 and an agent image 119 from a data storage device 115. The execution system 106 executes the application image 118 as an instance 108 of the application image 118. The execution system 106 starts the instances 104 and 105 of the agent image 119 from the agent image 119.

[0019] Application image 118 (and instance 108) includes a launcher subsystem 112, a multiplexer subsystem 116, and an application 117. Each of instances 104, 105 of agent image 119 includes a proxy service 107 and an application 109. Multiplexer subsystem 116 launches instances 104, 105 of agent image 119 by transmitting a request to interface 111. In this example, launcher subsystem 112 requests to launch multiple instances of agent image 119. In this example, interface 111 is an application programming interface (API) server. In response, the API server launches instances 104, 105 of agent image 119. Each of instances 104, 105 includes a proxy service 107 and an application 109, which is an application of the same type as application 117 (e.g., application 109 may be the same as application 117 and / or application 109 may include the same modules as application 117). The launcher subsystem 112 communicates with the proxy service 107 to instruct the proxy service 107 which parts of the application 109 to be executed by a particular instance of the proxy image 119. The proxy service 107 in turn instructs the application 109 which parts (or components) to execute.

[0020] Application 117 is a data processing graph (giving an example of a computer program) comprising vertices (representing data processing components (e.g., executable components) or data sets) connected by directed links (representing the flow of work elements, i.e., data) between vertices. In addition to these data flow connections, some data processing graphs also have control flow connections for determining the flow of control (e.g., control flow) between components. In such data processing graphs, program parts are components, and these program parts are interconnected according to their data flow links. In other examples, program parts are submodules or other entities within a program that are individually granted computing resources for execution. Program parts are considered to be interrelated to the extent that the execution capability of the entire program to which they belong depends on the capabilities of the individual program parts. Such interrelated or interdependent program parts can also be executed interdependently. For example, a program part can receive data from another program part or provide data to another program part. In addition, although program parts are individually granted computing resources, these program parts can overlap or be interdependent in various other ways (e.g., competing for computing resources with a limited supply).

[0021] Such an environment for developing graph-based computations is described in more detail, for example, in U.S. Publication No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "EXECUTING COMPUTATIONS EXPRESSED AS GRAPHS," which is incorporated herein by reference. A data processing graph made according to this system provides methods for inputting information into and out of individual processes represented by graph components, for moving information between processes, and for defining an order in which processes run. The system includes an algorithm for selecting an inter-process communication method from any available method (e.g., the communication paths according to the links of the graph may use TCP / IP or UNIX domain sockets, or use shared memory to pass data between processes).

[0022] Instance 108 of application image 118 includes application 117 (which is the same as application 109) to enable launcher subsystem 112 to identify the various components or parts of application 117 and how they communicate with each other to enable instance 108 to instruct instances 104, 105 how to communicate with each other (e.g., when a component executed by instance 105 requires input data output by a component executed by instance 104). In this example, host system 120 is configured to communicate with execution system 106, e.g., so that execution system 106 can launch instances 104, 105 of container image 119 on one or more of nodes 120a to 120r. In this example, host system 120 is a Kubernetes container that hosts and runs (e.g., executes) instances of container image 119. system.

[0023] refer to Figures 2A to 2F , shows a time series of the dynamic distribution of container image instances. Figure 2A , environment 101 is environment 100 at the first time ("T1") ( Figure 1 ) example. At time T1, the execution system 106 retrieves an instance 108 of the application image 118 from the data storage device 115. As previously described, the application image 118 is a container image that includes a launcher subsystem, a multiplexer subsystem, and an application 117. The application 117 is a program with three modules (modules 1 to 3). Each module specifies the amount of parallelism to be implemented in the module. In this example, the application 117 includes a specification that specifies the number of parallel executions for a given module, and the specification is accessed by the launcher subsystem 112, for example, when executing the configuration. Generally, the amount of parallelism refers to the number of times a particular module is executed, such as at the same time, simultaneously, etc. Module 1 is configured to execute three times in parallel. Module 2 is configured to execute twice in parallel. Module 3 is configured to execute three times in parallel. Reference Figure 2B , shows environment 103. Environment 103 is environment 101 ( Figure 1 ) at a second time ("T2"). In this example, the launcher subsystem 112 transmits multiple instances of the container image to be started for each module to the multiplexer subsystem according to the specified parallelism. The launcher subsystem 112 instructs the multiplexer subsystem 116 to start three instances 122a to 122c of the container image 119 and configure these instances 122a to 122c to execute module one. Because the parallelism of module one is configured to be three, the launcher subsystem 112 instructs the multiplexer subsystem 116 to start three instances 122a to 122c of the container image 119. In response, the multiplexer subsystem 116 requests the interface 111 to start the instances 122a to 122c, and the interface 111 starts the instances 122a to 122c.

[0024] The launcher subsystem 112 instructs the multiplexer subsystem 116 to launch three instances 124 a and 124 b of the container image 119 and configure these instances 124 a and 124 b to execute module 2. Because the parallelism of module 2 is configured to be two, the launcher subsystem 112 instructs the multiplexer subsystem 116 to launch two instances 124 a and 124 b of the container image 119, wherein each of the instances 124 a and 124 b is configured to execute module 2. In response, the multiplexer subsystem 116 requests the interface 111 to launch the instances 124 a and 124 b, and the interface 111 launches the instances 124 a and 124 b in the execution system 106.

[0025] Launcher subsystem 112 instructs multiplexer subsystem 116 to launch three instances 126 a through 126 c of container image 119 and configures these instances 126 a through 126 c to execute module 1. Because the parallelism of module 1 is configured to be three, launcher subsystem 112 instructs multiplexer subsystem 116 to launch three instances 126 a through 126 c of container image 119. In response, multiplexer subsystem 116 requests interface 111 to launch instances 126 a through 126 c, and interface 111 launches instances 126 a through 126 c.

[0026] refer to Figure 2C , shows environment 113. Environment 113 is environment 100 ( Figure 1 ) version at the third time point (T3). At time T3, the launcher subsystem 112 configures the container image 119 ( Figure 1 ). Each of instances 122a, 122b, 122c includes a proxy service and an application having three modules. Instance 122a includes a proxy service 128 and an application 130 having modules 130a to 130c. Each of applications 130, 132, 134 can be the same as application 117, and / or each of applications 130, 132, 134 can include the same modules as application 117. The launcher subsystem 112 configures each of the instances 122a, 122b, 122c of the container image 119 to execute module one (e.g., module 130a in application 130), for example, by transmitting an indication to each of instances 122a to 122c about which module the instance should execute. In this example, the launcher subsystem 112 identifies the amount of parallelism for each module by looking up the amount of parallelism specified in application 117.

[0027] The launcher subsystem 112 configures the container image 119 ( Figure 1). Each of instances 124a and 124b includes a proxy service and an application having three modules. Instance 124a includes a proxy service 131 and an application 132 having modules 132a to 132c. The launcher subsystem 112 configures each of instances 124a and 124b of the container image 119 to execute module two (e.g., module 132b in application 132), for example, by transmitting an instruction to each of instances 124a and 124b regarding which module the instance should execute.

[0028] The launcher subsystem 112 configures the container image 119 ( Figure 1 ). Each of instances 126a, 126b, 126c includes a proxy service and an application having three modules. Instance 126a includes a proxy service 133 and an application 134 having modules 134a through 134c. The launcher subsystem 112 configures each of instances 126a, 126b, 126c of the container image 119 to execute module three (e.g., module 134c in application 134), for example, by transmitting an instruction to each of instances 126a through 126c regarding which module the instance should execute.

[0029] refer to Figure 2D , shows environment 127. Environment 127 is environment 100 ( Figure 1 ) at a fourth time point (T4). At time T4, execution system 106 transfers or assigns an instance of the container image to host system 120 for execution on the host system. In this example, instance 122a is assigned to node 120d. Upon receiving instance 122a, node 120d hosts and executes container image 119 ( Figure 1 ) instance 122a, thereby instantiating the container. Instance 122b is assigned to node 120e. After receiving instance 122b, node 120e hosts and executes container image 119 ( Figure 1 ). Similarly, instances 122c, 124a, 124b, 126a, 126b, 126c are assigned to nodes 120f, 120g, 120i, 120n, 120o, and 120q, respectively. Upon receiving instances 122c, 124a, 124b, 126a, 126b, 126c, nodes 120f, 120g, 120i, 120n, 120o, and 120q (on host system 120) host and execute container image 119 ( Figure 1 ) corresponding instance.

[0030] refer to Figure 2E , shows environment 140. Environment 140 is environment 100 ( Figure 1) at a fifth time point (T5). At time T5, container image 122a is instantiated as container 160 on node 120d. Generally, a container image is instantiated as a container when a command included in a program or application running on a node (or other computing system) uses the container image as a template and generates a container from it that is subsequently executed on the node.

[0031] The instantiation of container image 122a generates a container 160 having a proxy service 162, which corresponds to (e.g., is equivalent to) the proxy service 128 included in container image 122a. In some examples, proxy service 128 is a proxy service image (i.e., an image of a proxy service). In this example, when container image 122a is instantiated, proxy service 162 is instantiated based on the proxy service image. The instantiation of container image 122a also generates an application 164 (included in container 160). In this example, application 164 includes modules 164a, 164b, and 164c. Application 164 corresponds to application 130 included in container image 122a. In some examples, application 130 is an application image (i.e., an image of an application). In this example, when container image 122a is instantiated, application 164 is instantiated based on the application image.

[0032] During execution of container 160, application 164 passes monitoring data 145 back to launcher subsystem 112, e.g., to enable launcher subsystem 112 to track the progress of application 164 and to track errors that may occur in application 164. Specifically, module 164a is configured to transmit monitoring data 145 to proxy service 162, which in turn communicates with launcher subsystem 112 and passes monitoring data 145 back to launcher subsystem 112. Launcher subsystem 112 uses monitoring data 145 to track when a particular module (e.g., 164a) has completed execution, e.g., so that launcher subsystem 112 can instruct the next module to execute (e.g., the next module on the same node or the next module on a different node).

[0033] refer to Figure 2F , shows an environment 150. The environment 150 is the environment 100 ( Figure 1 ) at a sixth point in time (T6). Environment 150 illustrates inter-node communication between containers 160, 170, and 176 (executing on a host system and instantiated based on container images 122a, 124a, and 126a, respectively).

[0034] In this example, container 170 includes a proxy service 172 and an application 174 (having modules 174a, 174b, 174c). In this example, application 174 is configured to execute module 174b according to the configuration of container image 124a, from which container 170 is instantiated.

[0035] In this example, container 176 includes proxy service 180 and application 178 (with modules 178a, 178b, 178c). In this example, application 178 is configured to execute module 178c according to the configuration of container image 126a, from which container 176 is instantiated. In this example, container 160 is running on node 120d ( Figure 2D ), which is the node to which the container image 122a is assigned. Container 172 is executed on node 120g ( Figure 2D ), which is the node to which the container image 124a is assigned. Container 176 is executed on node 120n ( Figure 2D ), which is the node to which the container image 126a is assigned.

[0036] In this example, the complete application is executed by the various modules of the application, which communicate with each other and pass the output of one module to the input of another module, thereby enabling the entire application to be executed across multiple machines. In this example, each module in the application 164, 174, 178 is configured to communicate with other modules (e.g., running on different nodes) or with the execution system 106.

[0037] In this example, container 160 is configured to execute module 164a to generate output 151 and transmit output 151 to module 174b of container 170. Container 170 is further configured such that when module 174b executes, output 152 of module 174 is input to module 178c of application 178 on container 176. Container 176 is further configured to execute module 178c to generate output 153 and transmit the output back to execution system 106, e.g., to enable the execution system to monitor which modules have been executed and which modules have completed execution. In this example, each of proxy services 162, 172, 174 is configured to communicate with launcher subsystem 112, e.g., to transmit monitoring data back to the launcher subsystem, thereby enabling launcher subsystem 112 to track the progress of container execution.

[0038] In this example, containers 160, 170, 176 are capable of automatically generating connections (e.g., pipes or communication channels) between each other. In this example, container 160 includes a local data store 160a that includes the output address of module 164a. In this example, data store 160a stores the address of module 174b on container 170. During the assignment of a container to a node, host system 120 transmits the address of the node to which the instance is assigned to the execution system 106. The launcher subsystem 112 (or more generally, the execution system 106) then identifies which modules need to transmit information to other modules based on, for example, the content of application 117. In other words, the launcher subsystem identifies upstream modules and downstream modules. Using the received address information, the launcher subsystem 112 transmits the address of the module to which the output data is to be transmitted to the local data store.

[0039] In this example, containers 160, 170, and 176 include local data stores 160a, 170a, and 176a, respectively. Launcher subsystem 112 transmits the address of module 174b to local data store 160a, transmits the address of module 178c to local data store 170a, and transmits instructions to local data store 176a to transfer output data back to execution system 106.

[0040] After execution is complete, module 164a searches local data storage 160a for the address of a data structure to output module 164a's output. The found address specifies module 174b. Using this address, module 164a transmits output 151 to module 174b via link 151a. In some examples, module 164a establishes link 151a, for example, by establishing a communication channel to module 174b. By dynamically establishing link 151a once module 164a has completed execution, delays in starting execution of module 174b are reduced.

[0041] After execution is complete, module 174b looks up the address of a data structure in local data storage 170a to output module 174b's output. The found address specifies module 178c. Using this address, module 174b transmits output 152 to module 178c via link 152a. In some examples, module 174b establishes link 152a, for example, by establishing a communication channel to module 178c. By dynamically establishing link 152a once module 174b has completed execution, delays in starting execution of module 178c are reduced.

[0042] After execution is complete, module 178c looks up the address of a data structure in local data storage 176a to output the output of module 178c. The found address specifies execution system 106. Using this address, module 178c transmits output 153 to execution system 106 via link 153a. In some examples, module 178c establishes link 153a, for example, by establishing a communication channel to the execution system.

[0043] In this example, module 164a can look up the address of output 151 from data storage 160a, for example, at the start-up time of module 164a. In another example, module 164a looks up the address while processing data. Based on these lookups, modules 164a, 174b, and 178c can, for example, stream data continuously and in real time without having to store the data on disk. In prior art systems, data must be stored on disk to repartition the data. In contrast, the system described herein can repartition data continuously and in real time by being able to look up the addresses of output nodes, without having to store the data on disk.

[0044] refer to Figure 3A , data processing diagram 200 includes components 202 through 210. Data processing diagram 200 also includes portions or groupings of components. Data processing diagram 200 includes portion 212 and portion 214. Portion 212 includes components 202 through 206. Portion 214 includes components 208 and 210. Portion 212 operates in parallel in two modes, and portion 214 operates in parallel in four modes.

[0045] refer to Figure 3B , FIG. 201 shows portions 212, 214 ( Figure 3A ) parallel execution. Application mirror 118 ( Figure 1 ) instance 108 starts the proxy image 119 ( Figure 1 ) to execute the portion 212 ( Figure 3A The launcher subsystem instructs each of the instances 216, 218 to execute the components 202, 204, 206. The application image 118 ( Figure 1 ) instance 108 starts the proxy image 119 ( Figure 1 ) to execute the portion 214 ( Figure 3A ). The launcher subsystem instructs each of the instances 220 to 226 to execute the components 208, 210.

[0046] Computing resource allocation

[0047] refer to Figure 4A, environment 300 includes execution system 106, data source 302, and host cluster 306 for processing data from data source 302 using computing resources of host cluster 306, the host cluster including computing resources that may be distributed across multiple hosts (e.g., computing clusters such as servers). Figure 4A In FIG, there are three hosts: a first host H1 336, a second host H2 338, and a third host H3 340. Each host includes a finite amount of computing resources, which together comprise the total computing resources of the host cluster 306. Examples of computing resources managed and allocated by the execution system 106 may include any of the following: a usage share of a host processor (e.g., virtual cores designated as mapped to physical cores of one or more multi-core processors), a portion of a host's volatile memory (e.g., an amount of a designated host's main memory space), a portion of a host's non-volatile memory (e.g., an amount designated as a host's hard drive storage space), or a usage share of a communication channel (e.g., a portion of the bandwidth of a host's Ethernet interface). A single unit of computing resources may include multiple types of resources, such as a specified number of CPUs or virtual cores and a specified amount of main memory.

[0048] Execution system 106 includes resource request module 322. Host cluster 306 includes resource manager 320. Resource request module 322 interacts with resource manager 320 to allocate computing resources to components so that no one component is allocated more computing resources than it needs when another component lacks computing resources.

[0049] To simplify the explanation of the computing resource allocation method described herein, the computing resources of the host are represented as computing resource units (shown as squares within the host), which are all shown as having the same granularity (i.e., the minimum size that can be authorized). However, it should be noted that computing resources are not necessarily divided into units of fixed and equal granularity, but can be divided into units of various granularities or can be allocated using other alternative methods. In addition, to simplify the explanation of the computing resource allocation method described herein, Figure 4A All hosts in host cluster 306 are shown as having the same number of computing resource units, ie, 16. However, it should be noted that in general different hosts may have different amounts of computing resources.

[0050] The resource manager 320 receives requests for computing resources and grants or denies these requests based on the amount of available computing resources in the hosts of the host cluster 306. An example of such a resource manager 320 includes the "Hadoop YARN" resource manager, which is capable of receiving requests for computing resources for executing a computer program (or a portion of a program) and, if sufficient computing resources are available, granting a "workspace" with a certain number of computing resource units for use by the program, wherein the workspace can be implemented as any suitable data structure that is used to contain a specific amount of computing resources, or contains any information that identifies a specific amount of computing resources, or any combination thereof. The computer program can then use the computing resources in the granted workspace to execute. In some examples, a computer program can request multiple resource workspaces (e.g., multiple workspaces for concurrent instances of a portion of a program) from the host cluster 306 at once. If there are sufficient resources available for the resource manager 320 to grant all requested multiple workspaces to the computer program, the resource manager will do so. Otherwise, based on available resources, the resource manager 320 may grant only some of the requested workspaces (i.e., an integer number of workspaces less than the total number of workspaces requested), or the resource manager 320 may not grant any of the requested workspaces. In some embodiments, all computing resources associated with a given workspace originate from a single host. Alternatively, in other embodiments, the resources for a given workspace may originate from multiple hosts.

[0051] As described in more detail below, resource request module 322 interacts with resource manager 320 in a manner that ensures that the multiple constraints imposed by data processing graph 200 are satisfied and / or satisfied in a more balanced manner.

[0052] As previously described, data processing graph 200 is a specification of a computer program for processing data received from data source 302. Data processing graph 200 includes a plurality of interrelated components including first component 202, second component 204, third component 206, fourth component 208, and fifth component 210.

[0053] Typically, each component of a data processing graph can be associated with one or more constraints. Such constraints can be provided by a "layout" that specifies constraints related to the parallelization of the component. One of the constraints specified in the layout is a "layout type", which can take one of a number of values ​​including a fixed depth dynamic layout (FDL) type. The different layout types specify constraints related to the number of instances of the component that are executed in parallel when the component is executed. Each of these component instances will consume computational resources, so the target amount of computational resources required for a component with a particular layout is directly determined by the corresponding target number of component instances for that component.

[0054] The dynamic layout type (FDL) can assign an instance to execute on a different host than the host storing the data that the instance is to operate on, which can provide increased flexibility, but only at the cost of reduced locality. However, if locality is not important for a particular computation, the trade-off may be worthwhile.

[0055] Components with FDL type have a predefined fixed target number of component instances required to execute on host cluster 306 for data processing graph 200 to successfully run. There is no restriction on where (i.e., on which hosts in host cluster 306) component instances with FDL type execute.

[0056] A potential advantage of a dynamic placement type (e.g., FDL) is the flexibility of being able to start a computation even if there are no computation resources available on a particular (e.g., local) host, as long as there are computation resources available on some host in the cluster. Another potential advantage is that if a computation fails due to a failure of one or more hosts, the computation may be able to be restarted on a different set of hosts.

[0057] Another potential advantage of the FDL type is the ability to fix the degree of parallelism (ie, the number of instances) to limit startup time delays caused by waiting for requests with greater parallelism to be satisfied.

[0058] exist Figure 4A In the exemplary data processing diagram 200 of FIG, each of the components 202 to 210 has an FDL type. Components 202 to 206 each require two component instances to be run on host cluster 306, with no restrictions on where (i.e., on which hosts of host cluster 306) the component instances are run. Each of components 208, 210 requires four component instances to be run on host cluster 306, with no restrictions on where (i.e., on which hosts of host cluster 306) the component instances are run. As described herein, the first portion of the data processing diagram 200 includes components 202 to 206, and this first portion runs in parallel in two ways, thus requiring components 202 to 206 to run in two instances 324, 326. In this example, ( Figure 1 ) instances 324, 326 of the proxy image in the application image 118 ( Figure 1 )'s instance 108( Figure 1 ) starts. For section 212 ( Figure 3A ) starts two instances 324, 326, because part 212 ( Figure 3A) need to run in parallel in two ways. Each of the instances 324, 326 only executes components 202 to 206. The second part of the data processing diagram 200 includes components 208, 210, and this second part runs in parallel in four ways, so it is necessary for components 208, 210 to run in four instances 370 to 376 (as shown in FIG. Figure 4B shown).

[0059] Each of the components in the data processing diagram 200 is configured to process a data element (e.g., a block, file, or record) from a data set in the data source 302. In some examples, a block is a portion of data in a file that includes one or more records. Figure 3A 、 Figure 3B , first component 202 processes a first data element 352 from a first data set 342, second component 204 processes a second data element 354 from a second data set 344, third component 206 processes a third data element 356 from a third data set 346, and fourth component 208 processes a fourth data element 358 from a fourth data set 348. In the example described below, a portion of execution system 106 analyzes the data sets associated with each component to determine a target number of instances of the different components based on the number of data elements, and therefore a target amount of computing resources. However, in other examples, the target number of computing resources is not based solely on the number of data in the associated data sets, but may instead (or in addition) depend on other factors, such as explicit or implicit user demand, historical or estimated performance data, or a combination thereof. It should be noted that it is not required that each component be associated with a different data set. For example, many components may all process data from one or more of the same data sets.

[0060] Resource Allocation

[0061] Given the placement constraints associated with the components in data processing graph 200 , data processing graph 200 can run successfully only if two instances of each of components 202 - 206 execute on host cluster 306 and four instances of each of components 208 - 210 execute on host cluster 306 .

[0062] If resource request module 322 knew the resources required by the total number of instances of the components of data processing graph 200 and the resources available in host cluster 306, then the solution to the packing optimization problem could be used to perform the task of allocating computing resources to data processing graph 200 via resource manager 320. However, typically, resource request module 322 does not know the resources available in host cluster 306. Instead, resource request module 322 simply requests resources from resource manager 320 and waits to see whether the requested resources are granted.

[0063] refer to Figures 4A to 4B , execution system 106 prepares to execute data processing graph 200. Resource request module 322 analyzes the portions and components of data processing graph 200 to determine the resource requirements of each of the components. Resource request module 322 determines that each instance of first component 202 requires five units of computing resources, each instance of second component 204 requires three units of computing resources, each instance of third component 206 requires four units of computing resources, each instance of fourth component 208 requires five units of computing resources, and each instance of fifth component 210 requires two units of computing resources.

[0064] refer to Figure 4A Resource request module 322 interacts with resource manager 320 to allocate resources for instances 324, 326 of a first portion of data processing graph 200, where the first portion includes components 202, 204, 206. To this end, resource request module 322 sends a request 350 to resource manager 320 for two workspaces (i.e., one for each required instance of the portion including components 202, 204, 206) from hosts 336, 338, 340, each workspace having a size of twelve computing resource units. Resource manager 320 allocates nine computing resource units on first host 336, eight computing resource units on second host 338, and seven computing resource units on third host 340. Resource manager 320 responds with a message 352 indicating that the resource manager is able to allocate the requested computing resources and therefore satisfies request 350. For both workspaces, each workspace includes twelve computing resource units allocated for the first component 202 , the second component 204 , and the third component 206 , which are 100% satisfied as required for the data processing graph 200 to execute.

[0065] refer to Figure 4BResource request module 322 interacts with resource manager 320 to allocate resources for instances 370, 372, 374, and 376 of the second portion of data processing graph 200 (including components 208 and 210). To do this, resource request module 322 sends a request 360 to resource manager 320 from any host for four workspaces (i.e., one for each required instance of the portion), each workspace having a size of five computing resource units (which is the amount of computing resource units required to execute components 208 and 210 together). Resource manager 320 allocates four computing resource units on first host 336, seven computing resource units on second host 338, and nine computing resource units on third host 340. Resource manager 320 then responds with a message 362 indicating that it was able to allocate the requested computing resources and, therefore, fulfilled request 360. For each of the four workspaces, which includes five computing resource units allocated for the second portion of data processing graph 200, the second portion is 100% fulfilled, as required for execution of data processing graph 200.

[0066] refer to Figure 5 , shows a process 400 for launching multiple instances of a specified image according to a dynamic layout, wherein the specified image includes a multi-component program (e.g., a data processing graph) having multiple executable components, and the dynamic layout specifies, for each of one or more of the executable components, an amount of parallelism to implement when executing the executable component.

[0067] In this example, process 400 is implemented by a data processing system (e.g., execution system 106) for causing instances of a container image to execute on multiple host systems, wherein each container image includes a first application having multiple modules, and wherein the instances are configured to execute a given module according to a determined number of parallel executions of the given module. In operation, the data processing system retrieves data from data storage device 115 ( Figure 1 )Access (402) container image 119 ( Figure 1 ), the container image includes a first application (e.g., Figure 1 109 in the application) and a second application (e.g., Figure 1 , wherein the second application causes the data processing system to communicate with the execution instance (e.g., Figure 1 104 or 105) of a host system (e.g., Figure 1 The data processing system determines (404) a given module of the first application (e.g., Figure 2C The number of parallel executions of module 130a) in.

[0068] For a given module, the data processing system generates (406) one or more instances of the container image according to the determined number of parallel executions ( Figure 2C 122a through 122c in the example embodiment, wherein each instance includes a first application and a second application. For each instance, the data processing system configures (408) the instance to execute a given module of the first application, for example, by transmitting an instruction from the launcher subsystem 112 to the instance 122a regarding which module to execute.

[0069] The data processing system causes (410) each of the plurality of configured instances to execute on one or more of the host systems. For at least one of the plurality of configured instances, the data processing system causes (412) communication between the data processing system and one or more of the host systems executing the configured instance via a second application program of the configured instance.

[0070] Elastic handling with containers and dynamic layouts

[0071] The technology described herein enables elastic processing using containers and dynamic layouts. Specifically, over time, there has been a trend toward increased virtualization, and to some extent, the decoupling of compute from disk location. Consequently, there may be a cluster of servers, and applications may need to be executed where resources are idle. For example, a server ("server A") may be busy because an application has been running on server A for a long time today, but there is an empty server in the cluster ("server B"). In this example, the application will be running on server B today. However, another application running on server B may run for a long time tomorrow. Therefore, tomorrow, the application runs on server A. In a static configuration, the configuration of which servers execute which application is static or specified in advance. In a dynamic configuration or dynamic layout, the configuration of which servers execute which application and when is flexible, depending on the available resources. The combination of executable and container technologies provides the ability to run scalable distributed parallel applications on resources that are available when the application starts.

[0072] Containers are fundamentally mechanisms for controlling what processes can see and access within the system. These can be used to implement performance strategies very similar to those achieved by running on virtual machines (VMs) through virtualization platforms. However, containers are much lighter than VMs. One of the issues with VMs is that when a VM runs, a full copy of the operating system runs, consuming gigabytes of disk space and, while running, gigabytes of memory. With container technology, the kernel (for example, the Linux kernel) is shared. The disk footprint is also shared. As a result, containers are more efficient than VMs. Using containers, I / O performance is significantly improved. Disk and memory footprints are reduced. Boot time is significantly reduced. Because applications run inside containers, the operating system is already up and running. If a VM is started, the application must wait for the operating system to boot. Containers provide a predictable environment. Therefore, what is tested in development is what runs in production. Containers also provide isolation. Therefore, if one application requires a patch or bug fix, the patch will not affect another application.

[0073] Scaling out across clusters

[0074] Data flow graphs (e.g., data processing graphs) and other execution applications are executed within these container platforms, whether a microservices architecture or a large-scale batch application is being implemented. When a container is running, the container only runs that graph. Therefore, for a services architecture, multiple instances of the container are implemented. Batch graphs can be run inside a single container, but the implementation will be limited to the resources available to the single container, and by definition, containers are limited to the resources available in the machine they are running on. This does not really provide the flexibility to scale out across as many available servers and provide as much computing power as is actually needed. Therefore, in order to scale out batch graphs and run them across clusters with a large number of cores and a large amount of available memory resources, a dynamic layout needs to be implemented.

[0075] Extended dynamic layout

[0076] Through dynamic layout, applications can be distributed and run in parallel across multiple servers. In addition, through container technology, applications can be packaged so that they are isolated. Through the technology described in this article, the system implements and utilizes batch graphs, containers, dynamic layouts, and scalable resources. For example, the graph and the operating system provide support for dynamic layouts that run using containers. Therefore, developers can write a graph and define a dynamic layout for the components in the graph, and specify a container infrastructure with a cluster of computing resources, and the graph should be executed across the cluster. Developers can simply insert the number of ways in which a graph or a specific component in the graph should run in parallel into the dynamic layout parameters of the graph. That is, instead of running containers and all logic and everything within one instance, the system is configured to execute multiple containers based on how many parallel ways the application is specified to run, and then enable the application to dynamically expand these containers (for example, execute across these containers).

[0077] At runtime, when this container appears, the system described in this article will interact with the orchestration platform (e.g., Kubernetes) , Marathon, YARN, or another scheduler) and ensures that the graph will execute across the entire cluster, that data will stream, and that the graph will utilize the available compute and memory resources in the cluster. All of this is achieved through dynamic layout parameters when configuring the dataflow graph. The combination of dynamic layout and containers enables multiple containers to execute a single application.

[0078] That is, through containerization technology and dynamic layout, the system described in this article can execute multiple instances of a container and leverage those instances to achieve scalability across batch applications. Through containers and dynamic layout, the system described in this article is fully scalable and dynamic, and has elastic computing capabilities independent of the actual presence of storage.

[0079] The technology described herein can be implemented, for example, using a programmable computing system that executes suitable software instructions, or can be implemented in suitable hardware such as a field programmable gate array (FPGA) or in some hybrid form. For example, in a program-controlled method, the software can include one or more computer programs that are executed on one or more program-controlled or programmable computing systems (which can be various architectures, such as distributed client / servers, or power grids), each of which includes at least one processor, at least one data storage system (including volatile and / or non-volatile memory and / or storage elements), at least one user interface (for receiving input using at least one input device or port, and for providing output using at least one output device or port). The software can include one or more modules of a larger program that, for example, provides services related to the design, configuration, and execution of a data flow graph. Program modules (e.g., elements of a data flow graph) can be implemented as data structures or other organized data that conform to a data model stored in a data repository.

[0080] The software may be stored in a non-transitory form, such as embodied in a volatile or non-volatile storage medium, or any other non-transitory medium, that utilizes the physical properties of the medium (e.g., surface pits and lands, magnetic domains, or charge) over an extended period of time (e.g., the time between refresh cycles of a dynamic memory device such as dynamic RAM). In preparation for loading the instructions included in the software stored in a non-transitory form, the software may be provided on a tangible, non-transitory medium such as a CD-ROM or other computer-readable medium (e.g., readable by a general-purpose or special-purpose computing system or device), or may be delivered (e.g., encoded in a propagating signal) to a computing system on which it is executed via a communication medium of a network. Some or all of the processing may be performed on a dedicated computer, or using dedicated hardware such as a coprocessor or a field programmable gate array (FPGA) or a specialized application-specific integrated circuit (ASIC). The processing may be implemented in a distributed manner, with different portions of the computation specified by the software being performed by different computing elements. Each such computer program is preferably stored or downloaded onto a computer-readable storage medium (e.g., solid-state memory or media, or magnetic or optical media) of a storage device accessible by a general or special purpose programmable computer to configure and operate the computer when the computer reads the storage device media to perform the processes described herein. The system of the present invention may also be considered to be implemented as a tangible, non-transitory medium configured with a computer program, wherein the medium so configured causes the computer to operate in a specific and predefined manner to perform one or more of the process steps described herein.

[0081] Several embodiments of the present invention have been described. However, it should be understood that the foregoing description is intended to illustrate, not to limit, the scope of the present invention, which is defined by the scope of the appended claims. Therefore, other embodiments are also within the scope of the appended claims. For example, various modifications may be made without departing from the scope of the present invention. In addition, some of the above steps may be sequence-independent and, therefore, may be performed in an order different from that described.

Claims

1. A method implemented by a data processing system for causing instances of a container image to execute on multiple host systems, wherein: Each container image includes a first application having a plurality of modules, and wherein the instances are configured to execute a given module according to a determined number of parallel executions of the given module, wherein the method comprises: accessing a container image that includes the first application and a second application, wherein the second application causes communication between the data processing system and a host system executing an instance of the container image; determining, by the data processing system, a number of parallel executions of a given module of the first application; For the given module, generating multiple instances of the container image according to the determined parallel execution quantity, wherein each instance includes the first application and the second application; For each instance, configuring the instance to execute a given module of the first application; causing each of the plurality of configured instances to execute on one or more of the host systems; and For at least one of the plurality of configured instances, a second application of the configured instance causes communication between the data processing system and one or more of the host systems executing the configured instance.

2. The method of claim 1 further comprising accessing a specification that specifies an amount of parallel execution for the given module.

3. The method according to claim 1, wherein The container image is a first container image, and the method further includes: Accessing an instance of a second container image from a hardware storage device, wherein the second container image includes the first application and specifies an amount of parallelism for each module in the second application; and The second container image is stored on the execution system.

4. The method of claim 3, further comprising: A request is transmitted from the instance of the second container image to an interface included in the execution system to generate multiple instances of the container image according to the determined parallel execution quantity.

5. The method of claim 1 , further comprising: Each of the plurality of configured instances is assigned to one or more of the host systems.

6. The method according to claim 5, wherein: The assignments are performed dynamically by assigning each configured instance at runtime, and the assignments are not predetermined.

7. The method according to claim 6, wherein: The first application is a data flow graph having a plurality of components, the method further comprising: For each component, generating multiple instances of the container image according to the number of parallel executions determined for the component; and Dynamically run the dataflow graph on multiple nodes to achieve dynamic levels of parallelism by: For each component, dynamically assigning the generated instances to one or more nodes of the host systems; and Each of the plurality of assigned instances is caused to execute on one or more nodes of the host systems.

8. The method of claim 1, wherein: Communications between the data processing system and one or more of the host systems executing the configured instance include: Monitoring data is transmitted by the given module to the second application, and the monitoring data is passed by the second application to the data processing system, wherein the monitoring data is configured to be used by the data processing system to track when the given module has completed execution so that the data processing system can instruct another module to execute.

9. The method of claim 1, wherein: The given module is a first module, wherein the generated plurality of instances are a plurality of first instances, and wherein the operations further comprise: determining, by the data processing system, a number of parallel executions of the second module of the first application; For the second module, generating a plurality of second instances of the container image according to the parallel execution quantity determined for the second module, wherein each second instance includes the first application and the second application; For each second instance, configuring the second instance to execute a second module of the first application; causing each of the plurality of configured second instances to execute on one or more of the host systems; and obtaining the address of one of these second instances from the local data store; and An output from one of the first instances is provided from one of the first instances to one of the second instances according to the fetched address.

10. A system for starting multiple instances of a specified image according to a dynamic layout, wherein: The specified image includes a multi-component program having a plurality of executable components, and the dynamic layout specifies, for each of one or more of the executable components, an amount of parallelism to be implemented when executing the executable component, the system comprising: a memory configured to store an application image and an agent image, wherein each of the application image and the agent image is configured to share a kernel of an operating system of a host computer of the image, wherein the application image includes a launcher process that requests launching of multiple instances of the agent image, wherein the agent image includes an agent process configured to communicate with the launcher process of the application image, and wherein the agent image further includes a multi-component program including a plurality of executable components; an execution system configured to retrieve the application image from a memory to generate an instance of the application image on the execution system, wherein the instance of the application image includes the launcher process, wherein the execution system is further configured to request, through the launcher process, that the plurality of instances of the agent image be launched according to a specified dynamic layout, the specified dynamic layout specifying, for each of one or more of the executable components, an amount of parallelism to be implemented when executing the executable component across one or more of the plurality of hosts hosting the instances of the agent image; and An interface is configured to launch multiple instances of the proxy image according to the specified dynamic layout in response to receiving a request from an instance of the application image, wherein each proxy service of each launched instance of the proxy image is configured to communicate with the launcher process to obtain an indication of which executable component to execute.

11. The system of claim 10, wherein: This multi-component program is a data processing graph.

12. The system of claim 10, further comprising: A host cluster having a plurality of host systems, wherein each host system includes a plurality of computing resources; Among them, each executable component requires a specified amount of computing resources; wherein the dynamic layout specifies the number of times a portion of the multi-component program is executed; The host cluster is configured to allocate computing resources required for the portion of the multi-component program across one or more of the host systems according to the dynamic layout and according to computing resources required by the executable components in the portion.

13. The system of claim 10, wherein: The instance of the application image is a container, and wherein the instance of the agent image is a container.

14. The system of claim 10, further comprising: A host cluster, which is configured as follows: receiving an indication as to which portion of the multi-component program to execute; and Transmits data specifying the execution state.

15. The system of claim 10, wherein: Each of the instance of the application image and the instance of the agent image is configured to share a kernel with a host operating system.

16. One or more machine-readable hardware storage devices for enabling instances of a container image to be executed on multiple host systems, wherein: Each container image includes a first application having a plurality of modules, and wherein the instances are configured to execute a given module according to a determined number of parallel executions of the given module, wherein the one or more machine-readable hardware storage devices store instructions executable by a data processing system to perform operations comprising: accessing a container image that includes the first application and a second application, wherein the second application causes communication between the data processing system and a host system executing an instance of the container image; determining, by the data processing system, a number of parallel executions of a given module of the first application; For the given module, generating multiple instances of the container image according to the determined parallel execution quantity, wherein each instance includes the first application and the second application; For each instance, configuring the instance to execute a given module of the first application; causing each of the plurality of configured instances to execute on one or more of the host systems; and For at least one of the plurality of configured instances, a second application of the configured instance causes communication between the data processing system and one or more of the host systems executing the configured instance.

17. The one or more machine-readable hardware storage devices of claim 16, wherein: These operations also include access to specifications that specify the amount of parallel execution for a given module.

18. The one or more machine-readable hardware storage devices of claim 16, wherein: The container image is a first container image, and wherein the operations further include: Accessing an instance of a second container image from a hardware storage device, wherein the second container image includes the first application and specifies an amount of parallelism for each module in the second application; and The second container image is stored on the execution system.

19. The one or more machine-readable hardware storage devices of claim 18, wherein: These operations also include: A request is transmitted from the instance of the second container image to an interface included in the execution system to generate multiple instances of the container image according to the determined number of parallel executions.

20. The one or more machine-readable hardware storage devices of claim 16, wherein: These operations also include: Each of the plurality of configured instances is assigned to one or more of the host systems.

21. The one or more machine-readable hardware storage devices of claim 20, wherein: Assignments are performed dynamically by assigning each configured instance at runtime, and the assignments are not predetermined.

22. The one or more machine-readable hardware storage devices of claim 21, wherein: The first application is a data flow graph having a plurality of components, wherein the operations further include: For each component, generating multiple instances of the container image according to the number of parallel executions determined for the component; and Dynamically run the dataflow graph on multiple nodes to achieve dynamic levels of parallelism by: For each component, dynamically assigning the generated instances to one or more nodes of the host systems; and Each of the plurality of assigned instances is caused to execute on one or more nodes of the host systems.

23. The one or more machine-readable hardware storage devices of claim 16, wherein: The given module is a first module, wherein the generated plurality of instances are a plurality of first instances, and wherein the operations further comprise: determining, by the data processing system, a number of parallel executions of the second module of the first application; For the second module, generating a plurality of second instances of the container image according to the parallel execution quantity determined for the second module, wherein each second instance includes the first application and the second application; For each second instance, configuring the second instance to execute a second module of the first application; causing each of the plurality of configured second instances to execute on one or more of the host systems; and A communication channel is established between one of the first instances and one of the second instances, wherein one of the first instances outputs data and transmits the output data to one of the second instances through the communication channel.

24. A data processing system for causing instances of a container image to execute on multiple host systems, wherein: Each container image includes a first application having a plurality of modules, and wherein the instances are configured to execute a given module according to a determined number of parallel executions of the given module, wherein the data processing system comprises: one or more processors; and One or more machine-readable hardware storage devices storing instructions executable by the one or more processors to perform operations including: accessing a container image that includes the first application and a second application, wherein the second application causes communication between the data processing system and a host system executing an instance of the container image; determining, by the data processing system, a number of parallel executions of a given module of the first application; For the given module, generating multiple instances of the container image according to the determined parallel execution quantity, wherein each instance includes the first application and the second application; For each instance, configuring the instance to execute a given module of the first application; causing each of the plurality of configured instances to execute on one or more of the host systems; and For at least one of the plurality of configured instances, a second application of the configured instance causes communication between the data processing system and one or more of the host systems executing the configured instance.

25. The data processing system of claim 24, wherein: These operations also include access to specifications that specify the amount of parallel execution for a given module.

26. The data processing system of claim 24, wherein: The container image is a first container image, and wherein the operations further include: Accessing an instance of a second container image from a hardware storage device, wherein the second container image includes the first application and specifies an amount of parallelism for each module in the second application; and The second container image is stored on the execution system.

27. The data processing system of claim 26, wherein the operations further comprise: A request is transmitted from the instance of the second container image to an interface included in the execution system to generate multiple instances of the container image according to the determined number of parallel executions.

28. The data processing system of claim 24, wherein the operations further comprise: Each of the plurality of configured instances is assigned to one or more of the host systems.

29. The data processing system of claim 28, wherein: Assignments are performed dynamically by assigning each configured instance at runtime, and the assignments are not predetermined.

30. The data processing system of claim 29, wherein: The first application is a data flow graph having multiple components, the operations also including: For each component, generating multiple instances of the container image according to the number of parallel executions determined for the component; and Dynamically run the dataflow graph on multiple nodes to achieve dynamic levels of parallelism by: For each component, dynamically assigning the generated instances to one or more nodes of the host systems; and Each of the plurality of assigned instances is caused to execute on one or more nodes of the host systems.

31. The data processing system of claim 24, wherein: Communications between the data processing system and one or more of the host systems executing the configured instance include: Monitoring data is transmitted by the given module to the second application, and the monitoring data is passed by the second application to the data processing system, wherein the monitoring data is configured to be used by the data processing system to track when the given module has completed execution so that the data processing system can instruct another module to execute.

32. The data processing system of claim 24, wherein: The given module is a first module, wherein the generated plurality of instances are a plurality of first instances, and wherein the operations further comprise: determining, by the data processing system, a number of parallel executions of the second module of the first application; For the second module, generating a plurality of second instances of the container image according to the parallel execution quantity determined for the second module, wherein each second instance includes the first application and the second application; For each second instance, configuring the second instance to execute a second module of the first application; causing each of the plurality of configured second instances to execute on one or more of the host systems; and obtaining the address of one of these second instances from the local data store; and An output from one of the first instances is provided from one of the first instances to one of the second instances according to the fetched address.

Images