A computer-implemented method, program product, and system for dynamically modifying task parallelism in a pipeline

Abstract

In a method of dynamically identifying and modifying the parallelism of specific tasks in a pipeline, the optimal execution time of each stage in the dynamic pipeline is calculated. The actual execution time of each stage in the dynamic pipeline is measured. It is determined whether the actual completion time of the data processing job will exceed a threshold. If it is determined that the actual completion time of the data processing job will exceed the threshold, an additional instance of the stage is created.

Description

BACKGROUND

[0001] The present invention relates generally to the field of using pipelines and more specifically to dynamically identifying and modifying parallelism of specific tasks in a pipeline.

[0002] In computing, a pipeline is a collection of data processing elements connected in series, where the output of one stage is the input of the next. A processing pipeline performs in concept in parallel, where the output stream of one stage is automatically fed as the input stream of the next stage. For example, the Linux system call pipeline is a classic example of this concept.

[0003] Since the throughput of a pipeline cannot be better than that of its slowest element, designers attempt to divide the work and resources among the stages so that each stage takes the same amount of time to complete its task. If a stage takes longer than the others, the designer can provide two or more processing elements to perform the stage in parallel, where a single input buffer feeds all instances of the stage and a single output buffer receives the results of all instances of the stage. When each instance of a pipeline stage completes processing of its current data item, it delivers the result to the common output buffer of the stage and fetches the next data item from the common input buffer of the stage.

[0004] The current state of the art has limitations. Typically, the number of partitions and parallelism of a task (or job or run) is determined at the start of execution. There has been some research into dynamically crossing more processes for critical resources. The problem is determining how much parallelism can improve performance without making the cost too high. For example, parallelism in a sort operation can not be very helpful, while for some other operations, additional parallelism would be helpful. Note that the latency often depends on the workload and thus often the optimal configuration cannot be predicted at the start of the job. Currently, there is no prior art to predict or suggest a good starting configuration for parallelism for a data pipeline. In the prior art, machine learning has not been utilized to optimize the use of parallelism. Another area that is not addressed at the prior art level is guaranteeing the quality of service (QoS) of a job, since many service level agreements (SLAs) include QoS terms that must be met. SUMMARY

[0005] Embodiments of the present invention disclose a method, computer program product, and system of dynamically identifying and modifying parallelism of specific tasks in a pipeline. In one embodiment, the optimal execution time of each stage in a dynamic pipeline is computed. The actual execution time of each stage in the dynamic pipeline is measured. It is determined whether the actual completion time of the data processing job will exceed a threshold. If it is determined that the actual completion time of the data processing job will exceed the threshold, an additional instance of the stage is created.

[0006] In one embodiment, it is determined whether sufficient resources are available to generate one or more new pipeline-level instances that negatively impact service quality. In response to determining that sufficient resources are available to generate one or more new pipeline-level instances, new pipeline-level instances are generated. Pipeline data is segmented into these new pipeline-level instances.

[0007] In one embodiment, the throughput of one or more new instances at one or more pipeline levels is measured. It is determined whether the throughput of the new instances at the pipeline level has increased. In response to determining that the throughput of the new instances at the pipeline level has increased, additional new instances at the pipeline level are created.

[0008] In one embodiment, the throughput of one or more new instances of one or more pipeline levels is measured. It is determined whether the throughput of the new instances at the pipeline level has increased. If it is determined that the throughput of the new instances at the pipeline level has not increased, the new instances at the pipeline level are removed. The current configuration of the dynamic pipeline used for the data processing job is stored. A decision tree model is trained to predict the behavior of the dynamic pipeline for the data processing job. Attached Figure Description

[0009] Figure 1 This is a functional block diagram illustrating a distributed data processing environment according to an embodiment of the present invention.

[0010] Figure 2 This is an example of a pipeline data flow model according to an embodiment of the present invention.

[0011] Figure 3a This is an example of the initial configuration of a three-stage pipeline in operation according to an embodiment of the present invention.

[0012] Figure 3b This is an example of an intermediate configuration of a three-stage pipeline in operation according to an embodiment of the present invention.

[0013] Figure 3c This is an example of another intermediate configuration of a three-stage pipeline in operation according to an embodiment of the present invention.

[0014] Figure 4 This is a flowchart depicting the operation steps of a dynamic pipeline procedure according to an embodiment of the present invention. Figure 1 Parallelism of specific tasks in a pipeline is dynamically identified and modified on computing devices within a distributed data processing environment.

[0015] Figure 5 It is a description of an embodiment according to the present invention. Figure 1 In a distributed data processing environment, computing devices on Figure 4 The flowchart of step 416 in the flowchart is a flowchart of the operation steps for intelligently generating additional instances to dynamically identify and modify the parallelism of specific tasks in the pipeline.

[0016] Figure 6 a distributed data processing environment in accordance with an embodiment of the present application. Figure 1 a block diagram of components of a computing device executing a dynamic pipeline program within a distributed data processing environment in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0017] A pipeline, also known as a data pipeline or data processing pipeline, is a set of data processing operations connected in series, where the output of one operation is the input of the next operation. The elements of the pipeline are often executed in parallel or in time-sliced fashion. A data processing workflow consists of several processing operations, where X stage processes the output of X-1 stage and passes its output to X+1 stage. The amount of parallelism is predefined for a particular workflow based on factors such as input load and SLA clauses.

[0018] The current state of the art has limitations. Typically, the number of partitions and parallelism of a task (or job or run) is decided at the beginning of execution. There has been some research on dynamically crossing more processes for critical resources. The problem is to determine how much parallelism can improve performance without making the cost too high. For example, parallelism in a sort operation can not be very helpful, while for some other operations, additional parallelism would help. In the prior art, machine learning has not been utilized to optimize the use of parallelism.

[0019] Another area not addressed in the state of the art is guaranteeing QoS of a job, as many SLAs include QoS clauses that must be met. For example, a common type of operation subject to QoS clauses in SLAs is an extract, transform, load (ETL) job, which is often time-critical. Note that the performance latency of intermediate stages is often based on the workload, so the optimal configuration cannot be predicted at the beginning of the job. If the time taken by the service exceeds the expected limit, dynamically rewriting the job (by increasing or decreasing parallelism) for the intermediate stages can result in increased performance to meet the QoS clause in the SLA.

[0020] Further, the prior art does not predict or suggest a good starting configuration for parallelism of a data pipeline. For example, a data quality (data processing) workflow takes longer than expected to find duplicate data in a job. This extra time threatens the QoS clause in the SLA. It is not known whether increasing the parallelism of the entire job, or just of a particular stage, would be beneficial. At the prior art level, the decision of how many parallel stages to use is made at the time the job is created, and cannot be changed. The solution is to use machine learning, whereby details of past runs can be accumulated to predict the amount of parallelism for each operation (within a task) to balance the increased performance and available resources. Based on past runs of these tasks, a machine learning model can be created to predict how a particular operation would behave if scaled up (more parallel instances added to the stage) or scaled down (parallel instances removed from the stage). Based on the predicted behavior, some of the operations can be scaled up or down to meet the QoS clause in the SLA. As a result, the parallelism of the job or stage is optimally modified without the need to restart as required in the prior art state. In a preferred embodiment, a data processing workflow is triggered, and the system will automatically modify the parallelism of each individual operation based on the availability of system resources, without regard to the parallelism of other operations in the workflow.

[0021] Figure 1 is a functional block diagram illustrating a distributed data processing environment (generally designated 100) suitable for operation of dynamic pipeline program 112 in accordance with at least one embodiment of the present application. The term "distributed" as used herein describes a computer system that includes multiple physically distinct devices that operate together as a single computer system. Figure 1 Only one implementation is provided for illustration and no implication is to be inferred that any other implementation is impracticable. Many modifications in addition to those described above can be made to the environments described herein without departing from the scope of the present application as recited in the claims.

[0022] The computing device 110 can be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In embodiments, the computing device 110 can be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating via a network 120 with other computing devices (not shown) within the distributed data processing environment 100. In another embodiment, the computing device 110 can represent a server computing system such as utilizing multiple computers as a server system in a cloud computing environment. In yet another embodiment, the computing device 110 represents a computing system utilizing clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within the distributed data processing environment 100.

[0023] In an embodiment, the computing device 110 includes a dynamic pipeline program 112. In embodiments, the dynamic pipeline program 112 is a program, application, or subprogram of a larger program for dynamically identifying and modifying parallelism of particular tasks in a pipeline in a distributed row-based database system. In alternative embodiments, the dynamic pipeline program 112 can be located on any other device accessible by the computing device 110 over the network 120.

[0024] In an embodiment, the computing device 110 includes an information store 114. In an embodiment, the information store 114 can be managed by the dynamic pipeline program 112. In alternative embodiments, the information store 114 can be managed separately by an operating system of the device, or in conjunction with the dynamic pipeline program 112. The information store 114 is a data store that can store, collect, compare, and / or combine information. In some embodiments, the information store 114 is located external to the computing device 110 and accessed through a communications network, such as the network 120. In some embodiments, the information store 114 is stored on the computing device 110. In some embodiments, the information store 114 can reside on another computing device (not shown) so long as the information store 114 is accessible by the computing device 110. The information store 114 includes, but is not limited to, client data, configuration data, database data, machine learning models, and other data received by the dynamic pipeline program 112 from one or more sources, and data created by the dynamic pipeline program 112.

[0025] As known in the art, information store 114 can be implemented using any volatile or non-volatile storage medium for storing information. For example, information store 114 can be implemented with a tape library, an optical library, one or more standalone hard drives, multiple hard drives in a redundant array of independent disks (RAID), a solid state drive (SSD), or random access memory (RAM). Similarly, information store 114 can be implemented with any suitable storage architecture known in the art, such as a relational database, a SQL database, a NoSQL database, an object-oriented database, or one or more tables.

[0026] Figure 2 is an example of a pipeline dataflow model, generally designated 200, in accordance with embodiments of the present application. In embodiments, a dynamic pipeline contains multiple stages. In the example illustrated in Figure 2 , the dynamic pipeline contains three stages, for illustration only. It should be noted that this figure is provided for purposes of illustration only. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present application.

[0027] Figure 2 includes pipeline 210, throughput monitor 220, resource monitor 230, CPU 240, RAM 250, and job execution optimization module 260.

[0028] In the embodiment illustrated in Figure 2 , pipeline 210 is an example of a section of a pipeline showing three stages of operations. In the illustrated embodiment, operation 1, operation 2, and operation 3 are three stages of operations of a longer pipeline. For example, operation 1 will receive its input from a previous pipeline stage (unless operation 1 is the first stage of the pipeline), while operation 3 will send its output to a subsequent pipeline stage (unless operation 3 is the last stage of the pipeline). In embodiments, there can be any number of stages in a pipeline.

[0029] In the embodiment illustrated in Figure 2 , throughput monitor 220 is an example of a module created by dynamic pipeline program 112 that monitors the stages of a pipeline to detect performance bottlenecks. The operation of throughput monitor 220 is explained below (see step 502 in Figure 5 ).

[0030] In the embodiment illustrated in Figure 2 , resource monitor 230 is an example of a module created by dynamic pipeline program 112 that monitors available resources, such as CPU 240 and RAM 250. The operation of resource monitor 230 is explained below (see step 504 in Figure 5 ).

[0031] InFigure 2 In the illustrated embodiment, CPU 240 is a sample processor in the system. In an embodiment, computing device 110 can have any number of processors, any of which involved in the operation of pipeline 210 are monitored by resource monitor 230. CPU 240 is representative of all processors monitored by resource monitor 230.

[0032] In the illustrated embodiment, RAM 250 is a sample memory in the system. In an embodiment, computing device 110 can have any number of RAM, or other forms of memory, any of which involved in the operation of pipeline 210 are monitored by resource monitor 230. RAM 250 is representative of all RAM or other forms of memory monitored by resource monitor 230. Figure 2 In the illustrated embodiment, job execution optimization module 260 is an instance of a module created by dynamic pipeline program 112 that takes the output of performance bottlenecks from throughput monitor 220 and resource availability data from resource monitor 230 and adds or subtracts levels of parallelism to each operation in the pipeline to maximize performance and meet QoS requirements. The operation of resource monitor 230 (e.g., all steps of

[0033] are controlled by job execution optimization module 260) is explained below. Figure 2 Figure 5 An example of the steps taken by dynamic pipeline program 112 to maximize performance and meet QoS requirements (generically designated as 300) according to an embodiment of the application is shown.

[0034] Figures 3a to 3c Is an example of the initial configuration of a three-stage pipeline with a single task per stage.

[0035] Figure 3a In the illustrated embodiment, dynamic pipeline program 112 has detected a slowdown in the pipeline at stage 2, and has therefore added a second instance to stage 2, now labeled pipeline 2a stage and pipeline 2b stage. In an embodiment, during normal data pipeline execution, job execution optimization module 260 determines that pipeline 2 stage is lagging in its processing based on input from throughput monitor 220. Job execution optimization module 260 checks with resource monitor 230 to determine if there are sufficient system resources available to support an additional instance of pipeline 2 stage. If so, job execution optimization module 260 creates an additional instance of stage 2 (pipeline 2b stage in this example).

[0036] In the illustrated embodiment, dynamic pipeline program 112 has detected a slowdown in the pipeline at stage 2, and has therefore added a second instance to stage 2, now labeled pipeline 2a stage and pipeline 2b stage. In an embodiment, during normal data pipeline execution, job execution optimization module 260 determines that pipeline 2 stage is lagging in its processing based on input from throughput monitor 220. Job execution optimization module 260 checks with resource monitor 230 to determine if there are sufficient system resources available to support an additional instance of pipeline 2 stage. If so, job execution optimization module 260 creates an additional instance of stage 2 (pipeline 2b stage in this example). Figure 3b

[0037] ​​A new instance of stage 2 is registered to allow stage 1 of the pipeline to send its output to both stage 2a and stage 2b of the pipeline. In an embodiment, the dynamic pipeline program 112 uses an algorithm to partition the output of stage 1 to distribute records from stage 1 substantially equally to the different instances of stage 2. For example, in an embodiment, the dynamic pipeline program 112 can use hashing to distribute the results of pipeline stage 1 to both pipeline stage 2a and pipeline stage 2b. Hashing divides a row by a hash value (an integer) and computes the modulus of the computed hash value with the number of rows to determine the number of partitions that should process the record. The hash value is computed in a random distribution manner to ensure that each partition receives approximately the same amount of records. In another example, the dynamic pipeline program 112 can use a round robin algorithm, where the first row is published to partition 1, the second row is published to partition 2,... N+1 row is published to partition 1, and so on. In another embodiment, the dynamic pipeline program 112 can use any other suitable algorithm that would be apparent to one skilled in the art to distribute the output from pipeline stage 1 to the two instances of pipeline stage 2 (i.e., stage 2a and stage 2b).

[0038] The new instance (pipeline stage 2b) begins to obtain data slices from pipeline stage 1, which reduces the processing load on pipeline stage 2a. If the throughput increases, this indicates that adding a new instance of pipeline stage 2 was the right choice. If higher performance is needed, the job execution optimization module 260 creates additional new instances of pipeline stage 2 in order to adjust performance as the processing load adjusts. Based on the improved performance of the pipeline, this configuration is stored as a good starting point for the next execution of this particular pipeline.

[0039] Based on the data collected above Figure 3b , the dynamic pipeline program 112 trains a decision tree model to predict the behavior of an operation based on any of the operation type, throughput, input operation, or output operation (which were captured above Figure 3b ). From this training, the dynamic pipeline program 112 predicts which operations would benefit from scaling up or down based on these parameters. Thus, rather than a random selection, the scaling up or down prediction would be based on the created model.

[0040] Figure 3c Further illustrating the example above Figure 3b . In Figure 3cIn this example, the dynamic pipeline program 112 has detected a slowdown in the pipeline at stage 3, and has therefore added a second instance to stage 3, now labeled pipeline 3a stage and pipeline 3b stage. In addition, the dynamic pipeline program 112 has detected that the pipeline at stage 2 now exceeds the throughput of the pipeline at stage 1 and stage 3, and therefore the dynamic pipeline program 112 has removed the second instance of stage 2 to free up resources for other tasks. In another example, the dynamic pipeline program 112 has determined based on input from the resource monitor 230 that the system needs more resources for other tasks, and therefore the dynamic pipeline program 112 has removed the second instance of stage 2 to free up resources for other tasks.

[0041] Figure 4 is a flowchart of a workflow 400 according to at least one embodiment of the present application, depicting the operational steps of the dynamic pipeline program 112 for seamlessly migrating data and indexes in a distributed row-based database system (e.g., SQL, NoSQL, or any row-based source) where data can be partitioned and processed in parallel. In alternative embodiments, the steps of the workflow 400 can be performed by any other program working in conjunction with the dynamic pipeline program 112. In an embodiment, the dynamic pipeline program 112 initializes the dynamic pipeline. In an embodiment, the dynamic pipeline program 112 records the throughput of each intermediate stage. In an embodiment, the dynamic pipeline program 112 builds a model to predict the completion time of each intermediate stage. In an embodiment, the dynamic pipeline program 112 identifies outliers. In an embodiment, the dynamic pipeline program 112 determines whether QoS will be impacted. In an embodiment, the dynamic pipeline program 112 creates a decision tree. In an embodiment, the dynamic pipeline program 112 predicts the combination of operations and parallelism that will benefit from creating additional instances. In an embodiment, the dynamic pipeline program 112 uses intelligent spawning of additional instances. In an embodiment, the dynamic pipeline program 112 uses a machine learning decision tree model to predict the optimal configuration of additional instances.

[0042] It should be appreciated that embodiments of the present application provide, at least, for dynamically identifying and modifying parallelism for specific tasks in a pipeline. However, Figure 4 Only one implementation is provided for the sake of illustration and is not intended to imply any limitation on the environments in which different embodiments can be implemented. Those skilled in the art will recognize many modifications to the described environments which fall within the scope of the present application as claimed.

[0043] The dynamic pipeline program 112 initializes the dynamic pipeline (step 402). At step 402, the dynamic pipeline program 112 creates the job execution optimization module 260, the throughput monitor 220, and the resource monitor 230 to initiate dynamic pipeline optimization. The job execution optimization module 260 performs actual additions and subtractions at the pipeline level based on inputs from the throughput monitor 220 and the resource monitor 230 to optimize throughput. The throughput monitor 220 continuously measures the throughput of each level in the pipeline and sends this status to the job execution optimization module 260. The resource monitor 230 continuously monitors the status of each resource required for the operation of the pipeline and sends the status to the job execution optimization module 260.

[0044] The dynamic pipeline program 112 records the throughput of each intermediate stage (step 404). At step 404, the dynamic pipeline program 112 uses the job execution optimization module 260 to record the throughput of each operation level in the pipeline based on inputs from the throughput monitor 220.

[0045] The dynamic pipeline program 112 builds a model to predict completion time (step 406). In an embodiment, the dynamic pipeline program 112 uses a machine learning algorithm to predict the expected completion time of each pipeline level and calculates the completion time of the entire job given the actual throughput of each level, the type of level, and the actual throughput since the start of the job. In an embodiment, the algorithm is a simple linear regression model. In another embodiment, the algorithm is a deep learning algorithm. In another embodiment, the dynamic pipeline program 112 can use any other suitable model apparent to those skilled in the art to predict the expected time of completion of a job based on the characteristics of each pipeline level or the job.

[0046] The dynamic pipeline program 112 identifies outliers (step 408). In step 408, the dynamic pipeline program 112 identifies levels where the predicted actual execution time is different from the expected execution time or from the minimum theoretically possible execution time. In an embodiment, the dynamic pipeline program 112 uses models to calculate the best execution time of each level that can be achieved under ideal conditions (i.e., if the throughput of the operation to which the input provided data is infinite). In an embodiment, these models can be machine learning models (e.g., other regression models). In another embodiment, these models can be simpler statistical formulas. In yet another embodiment, these models can be any other appropriate model apparent to those skilled in the art to identify levels where the predicted actual execution time is different from the expected execution time or from the minimum theoretically possible execution time.

[0047] In an embodiment, the dynamic pipeline program 112 uses the calculated gap between the predicted completion time of each stage and the stage running under ideal conditions to identify outlier data points. In an embodiment, an outlier data point is detected if the dynamic pipeline program 112 determines that the difference between the actual execution time of a stage (and thus job) and the expected execution time under ideal conditions for the same stage (and thus job) exceeds a threshold value. In an embodiment, the threshold value is a system default value. In another embodiment, the threshold value can be set by a user of the dynamic pipeline program 112.

[0048] The dynamic pipeline program 112 determines if QoS will be impacted? At step 410, if the dynamic pipeline program 112 determines that QoS will be impacted, the dynamic pipeline program 112 uses intelligent generation of intermediate operations to maintain QoS. The intelligent derivation process begins at step 412. If QoS will not be impacted, the dynamic pipeline program 112 ends.

[0049] The dynamic pipeline program 112 identifies the stage causing the bottleneck (step 412). At step 412, if the dynamic pipeline program 112 determines that QoS will be impacted, the dynamic pipeline program 112 determines which operation(s) in the stage should be generated in multiple instances to improve performance.

[0050] In an embodiment, the dynamic pipeline program 112 determines which stage is causing the bottleneck by using a simple heuristic that assumes that the first stage or the stage before in the flow that has a significant difference between predicted execution time and expected execution time is the most likely bottleneck. In another embodiment, the dynamic pipeline program 112 determines which stage is causing the bottleneck by identifying the pipeline stage where the output from the stage is less than the input to the stage. In yet another embodiment, the dynamic pipeline program 112 uses a more advanced algorithm that uses a machine learning model trained from past execution history of similar jobs to give more accurate predictions.

[0051] The dynamic pipeline program 112 predicts a combination of operations and parallelism (step 414). At step 414, the dynamic pipeline program 112 predicts the amount of parallelism that needs to be introduced. In an embodiment, the job execution optimization module 260 uses the output of step 412 to predict which combination of operations and desired parallelism can result in an optimal pipeline configuration to meet the QoS requirements. Based on the decision tree predictions, the job execution optimization module 260 intelligently generates or removes new instances of operations in the pipeline. In an embodiment, the dynamic pipeline program 112 predicts the amount of parallelism that needs to be introduced from the input to output ratio. In another embodiment, the dynamic pipeline program 112 predicts the amount of parallelism that needs to be introduced from past history of similar pipeline jobs. In an embodiment, the past history is determined from a machine learning model that is continuously retrained in both step 416 and step 518.

[0052] The dynamic pipeline program 112 uses the additional instance's intelligence to generate (step 416). At step 416, the dynamic pipeline program 112 uses the additional instance's intelligence on the stage identified in step 414 to generate. In an embodiment, the dynamic pipeline program 112 uses machine learning decision tree models to determine candidates of operations to generate. These decision tree models are created in step 406 and updated in this step and in the following step 518. If the dynamic pipeline program 112 determines that increasing the parallelism of a stage increases throughput, the dynamic pipeline program 112 feeds back the results to the machine learning decision tree models. At regular intervals, the dynamic pipeline program 112 re-trains those models with the latest historical results from both steps 416 and 518.

[0053] Details of the additional instance's intelligence generation are explained below in Figure 5 .

[0054] Figure 5 is a flowchart of a workflow 500 that describes the operational steps of the dynamic pipeline program 112 for intelligently generating additional instances according to at least one embodiment of the present application. In alternative embodiments, the steps of the workflow 500 can be performed by any other program when working with the dynamic pipeline program 112. In an embodiment, the dynamic pipeline program 112 identifies a limit operation stage X. In an embodiment, the dynamic pipeline program 112 determines if there are sufficient resources available. In an embodiment, the dynamic pipeline program 112 spawns a new stage X instance. In an embodiment, the dynamic pipeline program 112 partitions data to the new stage X instance. In an embodiment, the dynamic pipeline program 112 measures throughput. In an embodiment, the dynamic pipeline program 112 determines if the throughput has increased. In an embodiment, if the throughput has increased, the dynamic pipeline program 112 spawns an additional new stage X instance. In an embodiment, if the throughput has not increased, the dynamic pipeline program 112 removes the new stage X instance. In an embodiment, the dynamic pipeline program 112 stores the current configuration. In an embodiment, the dynamic pipeline program 112 trains decision tree models.

[0055] It should be appreciated that embodiments of the present application provide at least for dynamically identifying and modifying parallelism of specific tasks in a pipeline. However, Figure 5 Only one implementation is provided for the sake of illustration and is not intended to imply any limitation with regard to environments in which different embodiments can be implemented. Those skilled in the art will recognize that many modifications are possible in the described environments without departing from the scope of the present application as described in the claims.

[0056] The dynamic pipeline program 112 identifies a bottlenecked operation stage X (step 502). At step 502, the dynamic pipeline program 112 determines that stage X is lagging in processing. In an embodiment, the dynamic pipeline program 112 determines that stage X is lagging in processing using the job execution optimization module 260 based on input from the throughput monitor 220.

[0057] The dynamic pipeline program 112 determines whether sufficient resources are available (step 504). At step 504, the dynamic pipeline program 112 determines whether sufficient resources are available to spawn a new instance of stage X. In an embodiment, the dynamic pipeline program 112 determines whether sufficient resources are available using the job execution optimization module 260 based on input from the resource monitor 230. If the dynamic pipeline program 112 determines that sufficient resources are available to spawn a new instance of stage X, the dynamic pipeline program 112 proceeds to step 506. If the dynamic pipeline program 112 determines that sufficient resources are not available to spawn a new instance of stage X, the dynamic pipeline program 112 returns to the flow in FIG. 4. Figure 4 In an embodiment, sufficient resources include processor resources. In another embodiment, sufficient resources include memory resources. In yet another embodiment, sufficient resources include any combination of computer resources that can be required to create and manage a dynamic pipeline.

[0058] The dynamic pipeline program 112 spawns a new instance of stage X (step 506). At step 506, the dynamic pipeline program 112 spawns one or more new instances of stage X (scale up). In an embodiment, the dynamic pipeline program 112 uses the job execution optimization module 260 to add one or more new instances of stage X to an intermediate stage of the pipeline. In an embodiment, the number of new instances of stage X created by the dynamic pipeline program 112 is based on the performance calculations performed in step 414.

[0059] The dynamic pipeline program 112 partitions data to the new stage X (step 508). At step 508, the dynamic pipeline program 112 partitions data from stage X-1 to the new instances of stage X. In an embodiment, the dynamic pipeline program 112 uses the job execution optimization module 260 to partition data from stage X-1 to the new instances of stage X to increase parallelism of the intermediate stage. In the above Figure 3b Partitioning data to the new stage X is described below.

[0060] The dynamic pipeline program 112 measures throughput (step 510). At step 510, the dynamic pipeline program 112 measures the throughput of the overall pipeline to determine whether the new instances of stage X have increased throughput. In an embodiment, the dynamic pipeline program 112 determines whether the throughput of the pipeline has increased using the job execution optimization module 260 based on input from the throughput monitor 220.

[0061] The dynamic pipeline program 112 determines if throughput has increased (step 512). At step 512, the dynamic pipeline program 112 determines if the total throughput of the pipeline has increased. In an embodiment, the dynamic pipeline program 112 uses the job execution optimization module 260 to determine if the throughput of the pipeline has increased based on input from the throughput monitor 220. If the dynamic pipeline program 112 determines that the total throughput of the pipeline has increased, the dynamic pipeline program 112 returns to step 502 to determine if additional performance improvement is possible by adding additional parallel operations to the stage. If the dynamic pipeline program 112 determines that the total throughput of the pipeline has not increased, the dynamic pipeline program 112 continues to step 514.

[0062] The dynamic pipeline program 112 removes the new instance of the stage X (step 514). At step 514, if the dynamic pipeline program 112 determines that the total throughput of the pipeline has not increased, the dynamic pipeline program 112 removes the new instance of the stage X (scales down) to free up resources. In an embodiment, the dynamic pipeline program 112 uses the job execution optimization module 260 to remove the new instance of the stage X to free up resources.

[0063] The dynamic pipeline program 112 stores the current configuration (step 516). At step 516, the dynamic pipeline program 112 stores the current configuration of the dynamic pipeline to use as a possible starting point for the next run of the pipeline since the throughput has increased.

[0064] The dynamic pipeline program 112 trains the decision tree model (step 518). At step 518, based on the data from above, the dynamic pipeline program 112 trains the model to predict the behavior of operations based on operation types, throughput, input operations, output operations as captured from previous steps. Based on these parameters collected during the training, the dynamic pipeline program 112 predicts which operations would benefit from scaling up or down. In an embodiment, the dynamic pipeline program 112 uses a decision tree model to predict the behavior of the dynamic pipeline of a data processing job. Thus, future scaling up or down predictions would be based on the model created, rather than randomly selected. The dynamic pipeline program 112 then returns to step 502. Figure 4 .

[0065] Figure 6 is a block diagram depicting components of a computing device 110 suitable for use with the dynamic pipeline program 112, in accordance with at least one embodiment of the present application. Figure 6 The computer 600, one or more processors 604 (including one or more computer processors), a communication fabric 602, a memory 606 including a random access memory (RAM) 616 and a cache memory 618, a persistent storage 608, a communication unit 612, an I / O interface 614, a display 622, and external devices 620 are illustrated. It should be understood that Figure 6Only one embodiment is illustrated and described and it is not implied that an only one embodiment is implemented in an environment of any particular description. Numerous modifications can be made to the depicted environments.

[0066] As described, computer 600 operates in a communications structure 602 that provides communications between a computer processor 604, a memory 606, a persistent storage 608, a communications unit 612, and an input / output (I / O) interface 614. The communications structure 602 can be implemented with an architecture that is suitable for passing data or control information between the processor 604 (e.g., a microprocessor, a communications processor, and a network processor), the memory 606, the external device 620, and any other hardware components within the system. For example, the communications structure 602 can be implemented with one or more buses.

[0067] The memory 606 and the persistent storage 608 are computer readable storage media. In the depicted embodiment, the memory 606 includes RAM 616 and a cache 618. Generally, the memory 606 can include any suitable volatile or non-volatile computer readable storage media. The cache 618 is a fast memory that enhances the performance of the processor 604 by providing quick access to frequently accessed data and instructions.

[0068] The program instructions for the dynamic pipeline program 112 can be stored in the persistent storage 608, or more generally, in any computer readable storage media, for execution by one or more of the respective computer processors 604 via one or more memories of the memory 606. The persistent storage 608 can be a magnetic hard drive, a solid state disk drive, semiconductor memory, read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or any other computer readable storage media that can store program instructions or digital information.

[0069] The media used by the persistent storage 608 can also be removable. For example, a removable hard drive can be used for the persistent storage 608. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive, for transferring digital information to another computer readable storage media, which is also part of the persistent storage 608.

[0070] In these examples, the communications unit 612 provides communication through the use of physical and / or wireless communications links. In some embodiments of the application, the sources of the different input data can be physically remote from the computer 600, so that the input data can be received via the communications unit 612, and similarly, the output can be transmitted.

[0071] I / O interface 614 allows for input and output of data with other devices that can be connected to computer 600. For example, I / O interface 614 can provide a connection to external device 620 such as a keyboard, keypad, a touch screen, a microphone, a digital camera, and / or some other suitable input device. External device 620 can also include a portable computer readable storage medium such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data used to practice embodiments of the present application, for example, dynamic pipeline program 112, can be stored on such portable computer readable storage media and can be loaded onto permanent storage 608 via I / O interface 614. I / O interface 614 is also connected to display 622.

[0072] Display 622 provides a mechanism to display data to a user, and can be, for example, a computer monitor. Display 622 can also be used as a touch screen, such as the display of a tablet computer.

[0073] The programs described herein are based on applications identified as being implemented in particular embodiments of the present application. It should be understood, however, that any particular program terminology that can be used herein is used solely for convenience, and thus the present application should not be limited to only use in any particular applications identified and / or implied by such terminology.

[0074] The present application can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions stored therein (or thereon) for causing a processor to carry out aspects of the present application.

[0075] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire cable, because the computer readable storage medium is a tangible memory device that modulates or carries the computer readable instructions for performing, individually or in combination, one or more aspects of the present application.

[0076] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage and / or execution.

[0077] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or any combination of source code or object code in any combination of one or more programming languages including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on a user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0078] The computer readable program instructions can also be loaded onto a computing / processing device, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computing / processing device, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computing / processing device, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0079] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage viruses or other code or instructions implementing a functionally equivalent to that of the software manual reproduction process, i.e., a non-transitory computer readable storage medium that does not include a transitory signal per se. The instructions can be executed to create a means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0080] These computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0081] The flow diagrams and block diagrams in the drawings are representative of the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions ("instructions"). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0082] The descriptions of various embodiments of the present application have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present application. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application, or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer implementation method for dynamically modifying the parallelism of tasks in a pipeline, the computer implementation method comprising: The optimal execution time for each of the multiple stages of a dynamic pipeline used for a data processing job is calculated by one or more computer processors. The actual execution time of each of the plurality of stages of the dynamic pipeline used for the data processing job is determined by one or more computer processors. One or more computer processors determine whether the actual completion time of the data processing job will exceed a threshold based on the actual execution time of each of the plurality of stages of the dynamic pipeline used for the data processing job; as well as In response to determining that the actual completion time of the data processing job will exceed the threshold, one or more computer processors create one or more additional instances of one or more levels of the multiple levels of the dynamic pipeline for the data processing job, wherein the one or more additional instances of the multiple levels of the dynamic pipeline for the data processing job reduce the actual completion time of the data processing job. The creation of one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job includes: In response to determining that sufficient resources are available for generating the first or more additional instances, the first or more additional instances are generated; and In response to the determination by measurement that the throughput of the first one or more additional instances has not increased, the first one or more additional instances are removed; The machine learning decision tree model is trained to predict the behavior of the dynamic pipeline.

2. The computer implementation method according to claim 1, wherein, The method involves one or more computer processors determining whether the actual completion time of the data processing job will exceed the threshold based on the actual execution time of each of the plurality of stages of the dynamic pipeline used for the data processing job, and also includes using a machine learning model to predict the actual completion time of the data processing job.

3. The computer-implemented method according to any one of the preceding claims further comprises: One or more outlier data points are identified by one or more computer processors, wherein the one or more outlier data points identify pipeline stages in the plurality of stages of the dynamic pipeline of the data processing job where the actual execution time differs from the optimal execution time; as well as One or more computer processors determine, based on the one or more outlier data points, whether the predicted completion time of the data processing job will exceed the threshold.

4. The computer implementation method according to claim 1, wherein, The one or more additional instances of the one or more stages of the dynamic pipeline created by one or more computer processors for the data processing job further include: One or more computer processors determine whether sufficient resources are available to generate first or more additional instances of one or more of the multiple stages of the dynamic pipeline for the data processing job; and One or more computer processors divide pipeline data into the first or more additional instances of one or more of the multiple stages of the dynamic pipeline used for the data processing job.

5. The computer implementation method according to claim 1, wherein, The one or more additional instances of the one or more stages of the plurality of stages for creating the dynamic pipeline for the data processing job by one or more computer processors further include: The throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline used for the data processing job is measured by one or more computer processors. One or more computer processors determine whether the throughput of the first or more additional instances of the one or more stages of the dynamic pipeline for the data processing job has increased; and in response to determining that the throughput of the one or more additional instances of the one or more stages of the dynamic pipeline for the data processing job has increased, one or more computer processors generate a second or more additional instances of the one or more stages of the dynamic pipeline for the data processing job.

6. The computer implementation method according to claim 1, wherein, The one or more additional instances of the one or more stages of the plurality of stages for creating the dynamic pipeline for the data processing job by one or more computer processors further include: The throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline used for the data processing job is measured by one or more computer processors. One or more computer processors determine whether the throughput of the first or more additional instances of one or more of the plurality of stages of the dynamic pipeline used for the data processing job has increased; In response to removing the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job, one or more computer processors store the current configuration of the dynamic pipeline for the data processing job; and A decision tree model is trained by one or more computer processors to predict the behavior of the dynamic pipeline of the data processing job.

7. The computer implementation method according to claim 4, wherein, The creation of one or more additional instances of one or more levels of the dynamic pipeline for the data processing job by one or more computer processors also includes using a machine learning decision tree model to predict the optimal configuration of the one or more additional instances of the dynamic pipeline for the data processing job.

8. A computer program product for dynamically modifying the parallelism of tasks in a pipeline, the computer program product comprising: One or more computer-readable storage devices and program instructions stored on the one or more computer-readable storage devices, the stored program instructions including: Program instructions for calculating the optimal execution time for each of the multiple stages of a dynamic pipeline used for data processing jobs; Program instructions for determining the actual execution time of each of the plurality of stages of the dynamic pipeline for the data processing job; Program instructions for determining whether the actual completion time of the data processing job will exceed a threshold based on the actual execution time of each of the plurality of stages of the dynamic pipeline used for the data processing job; and In response to determining that the actual completion time of the data processing job will exceed the threshold, program instructions are provided to create one or more additional instances of one or more levels of the plurality of levels of the dynamic pipeline for the data processing job, wherein the one or more additional instances of the plurality of levels of the dynamic pipeline for the data processing job reduce the actual completion time of the data processing job. The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job include: In response to determining that sufficient resources are available for generating the first or more additional instances, program instructions for generating the first or more additional instances; and In response to the determination by measurement that the throughput of the first one or more additional instances has not increased, the program instruction to remove the first one or more additional instances is executed. The machine learning decision tree model is trained to predict the behavior of the dynamic pipeline.

9. The computer program product of claim 8, wherein the program instructions for determining whether the actual completion time of the data processing job will exceed the threshold based on the actual execution time of each of the plurality of stages of the dynamic pipeline of the data processing job further include: A machine learning model is used to predict the actual completion time of the data processing job.

10. The computer program product according to any one of claims 8 to 9, further comprising: Program instructions for identifying one or more outlier data points, wherein the one or more outlier data points identify pipeline levels in the plurality of levels of the dynamic pipeline of the data processing job whose actual execution time differs from the optimal execution time; as well as Program instructions for determining, based on the one or more outlier data points, whether the predicted completion time of the data processing job will exceed the threshold.

11. The computer program product according to claim 8, wherein, The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job further include: Program instructions for determining whether sufficient resources are available to generate a first or more additional instances of one or more of the multiple stages of the dynamic pipeline for the data processing job; and Program instructions for segmenting pipeline data into one or more additional instances of one or more of the plurality of levels of the dynamic pipeline used for the data processing job.

12. The computer program product according to claim 8, wherein, The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job further include: The throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline is measured for program instructions used in the data processing job. Program instructions for determining whether the throughput of the first or more additional instances of one or more of the plurality of stages of the dynamic pipeline used for the data processing job has been increased; and In response to determining that the throughput of the first or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job has increased, program instructions are given to generate a second or more additional instances of the one or more of the plurality of stages of the dynamic pipeline for the data processing job.

13. The computer program product according to claim 8, wherein, The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job further include: The throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline is measured for program instructions used in the data processing job. Program instructions for determining whether the throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline is used for the data processing job. In response to removing the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job, program instructions for storing the current configuration of the dynamic pipeline for the data processing job are provided; and The program instructions for training a decision tree model to predict the behavior of the dynamic pipeline used for the data processing job.

14. The computer program product according to claim 11, wherein, The program instructions for creating one or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job further include: using a machine learning decision tree model to predict the optimal configuration of one or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job.

15. A computer system for dynamically modifying the parallelism of tasks in a pipeline, the computer system comprising: One or more computer processors; One or more computer-readable storage media; as well as Program instructions stored on the one or more computer-readable storage media, for execution by at least one of the one or more computer processors, the stored program instructions including: Program instructions for calculating the optimal execution time for each of the multiple stages of a dynamic pipeline used for data processing jobs; Program instructions for determining the actual execution time of the data processing job for each of the plurality of stages of the dynamic pipeline; Program instructions for determining whether the actual completion time of the data processing job will exceed a threshold based on the actual execution time of each of the plurality of stages of the dynamic pipeline used for the data processing job; and In response to determining that the actual completion time of the data processing job will exceed the threshold, program instructions are provided to create one or more additional instances of one or more levels of the plurality of levels of the dynamic pipeline for the data processing job, wherein the one or more additional instances of the plurality of levels of the dynamic pipeline for the data processing job reduce the actual completion time of the data processing job. The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job include: In response to determining that sufficient resources are available for generating the first or more additional instances, program instructions for generating the first or more additional instances; and In response to the determination by measurement that the throughput of the first one or more additional instances has not increased, the program instruction to remove the first one or more additional instances is executed. The machine learning decision tree model is trained to predict the behavior of the dynamic pipeline.

16. The computer system according to claim 15, wherein, The program instructions for determining whether the actual completion time of the data processing job will exceed the threshold based on the actual execution time of each of the plurality of stages of the dynamic pipeline of the data processing job further include: using a machine learning model to predict the actual completion time of the data processing job.

17. The computer system according to any one of claims 15 to 16, further comprising: Program instructions for identifying one or more outlier data points, wherein the one or more outlier data points identify pipeline levels in the plurality of levels of the dynamic pipeline of the data processing job whose actual execution time differs from the optimal execution time; as well as Program instructions for determining, based on the one or more outlier data points, whether the predicted completion time of the data processing job will exceed the threshold.

18. The computer system according to claim 15, wherein, The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job further include: Program instructions for determining whether sufficient resources are available to generate a first or more additional instances of one or more of the multiple stages of the dynamic pipeline for the data processing job; and Program instructions for segmenting pipeline data into one or more additional instances of one or more of the plurality of levels of the dynamic pipeline used for the data processing job.

19. The computer system of claim 15, wherein the program instructions for creating one or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job further include: The throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline is measured for program instructions used in the data processing job. Program instructions for determining whether the throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline used for the data processing job has been increased; as well as In response to determining that the throughput of the first or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job has increased, program instructions are given to generate a second or more additional instances of the one or more of the plurality of stages of the dynamic pipeline for the data processing job.

20. The computer system according to claim 15, wherein, The program instructions for creating one or more additional instances of one or more of the plurality of levels of the dynamic pipeline for the data processing job further include: The throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline is measured for program instructions used in the data processing job. Program instructions for determining whether the throughput of the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline is used for the data processing job; and In response to removing the first one or more additional instances of one or more of the plurality of stages of the dynamic pipeline for the data processing job, program instructions for storing the current configuration of the dynamic pipeline for the data processing job are provided; and The program instructions for training a decision tree model to predict the behavior of the dynamic pipeline used for the data processing job.

Images