Method and system for proactive job allocation in multiple computing environments

Abstract

A method is disclosed and includes receiving, by a proactive job allocation system (PJAS) comprising a predictive core engine and a job allocator, from a scheduler, a plurality of jobs. The method also includes predicting, by the predictive core engine, probabilities of successful completion of the plurality of jobs with respect to a plurality of data systems based on historical information related to the plurality of jobs. The method also includes generating allocation decisions, by the predictive core engine, for allocating the plurality of jobs to the plurality of data systems based on the predicted probabilities of successful completion of the plurality of jobs. The method also includes distributing, by the job allocator, the plurality of jobs to the plurality of data systems based on the allocation decisions.

Description

PATENTAttorney Docket No. 079900-1456771 -8972WO01Client Ref. No. 8972WO01METHOD AND SYSTEM FOR PROACTIVE JOB ALLOCATION IN MULTIPLE COMPUTING ENVIRONMENTSBACKGROUND

[0001] Distribution of computational tasks among various data centers of computing environments may be challenging because of the dynamic nature of the computing environments. For example, rule-based strategies consider factors such as available resources and network latency. However, the resource availability and load conditions of those data centers may fluctuate and change often. Thus, the job performance may be impacted. As a result, such approaches may lead to suboptimal resource usage, increased job failure rates, and overall inefficiency across data centers. Thus, there is a need to address these challenges and others. Embodiments of the invention address these and other problems individually and collectively.BRIEF SUMMARY

[0002] One embodiment is related to a method comprising: receiving, by a proactive job allocation system (PJAS) comprising a predictive core engine and a job allocator, from a scheduler, a plurality of jobs; predicting, by the predictive core engine, probabilities of successful completion of the plurality of jobs with respect to a plurality of data systems based on historical information related to the plurality of jobs; generating allocation decisions, by the predictive core engine, for allocating the plurality of jobs to the plurality of data systems based on the predicted probabilities of successful completion of the plurality of jobs; and distributing, by the job allocator, the plurality of jobs to the plurality of data systems based on the allocation decisions.

[0003] Another embodiment is related to a non-transitory computer-readable medium storing computer-executable instructions to perform operations comprising: receiving, by a proactive job allocation system (PJAS) comprising a predictive core engine and a job1KILPATRICK TOWNSEND 78819593 1allocator, from a scheduler, a plurality of jobs; predicting, by the predictive core engine, probabilities of successful completion of the plurality of jobs with respect to a plurality of data systems based on historical information related to the plurality of jobs; generating allocation decisions, by the predictive core engine, for allocating the plurality of jobs to the plurality of data systems based on the predicted probabilities of successful completion of the plurality of jobs; and distributing, by the job allocator, the plurality of jobs to the plurality of data systems based on the allocation decisions.

[0004] Another embodiment is related to a system comprising: a predictive core engine; a job allocator; and a control unit; where the predictive core engine is configured to perform operations comprising predicting probabilities of successful completion of a plurality of jobs with respect to a plurality of data systems based on historical information related to the plurality of jobs; and generating allocation decisions for allocating the plurality of jobs to the plurality of data systems based on the predicted probabilities of successful completion of the plurality of jobs; and where the job allocator is configured to distribute the plurality of jobs to the plurality of data systems based on the allocation decisions.

[0005] Further details regarding embodiments of the disclosure can be found in the Detailed Description and the Figures.BRIEF DESCRIPTION OF THE DRAWINGS

[0006] FIG. 1 is a block diagram illustrating example computing environments 100 for computing job allocation, according to certain embodiments.

[0007] FIG. 2 is a block diagram illustrating an example architecture 200 of a proactive job allocation system (PJAS), according to certain embodiments.

[0008] FIG. 3 illustrates an example flowchart depicting processing performed by a proactive job allocation system (PJAS), according to certain embodiments.

[0009] FIG. 4 is a simplified block diagram of a reinforcement learning (RL) training environment 400 for training a predictive core engine (PCE) within PJAS, according to certain embodiments.2KILPATRICK TOWNSEND 78819593 1

[0010] FIG. 5 is a table illustrating an example reward function for reinforcement learning techniques, according to certain embodiments.

[0011] FIG. 6 is an example flowchart illustrating a method (e.g., RL) for training a predictive core engine (PCE) within PJAS, according to certain embodiments.DETAILED DESCRIPTION

[0012] The disclosed techniques describe an architecture, called a proactive job allocation system (PJAS), that can learn from historical information about computing environments, allocate a job to a computing environment, and distribute the job to that computing environment for execution, by taking into account the dynamic nature of the computing environments, to achieve optimal result of the job submission (i.e., allocation and distribution) and execution. The optimal result may include, but is not limited to, shorter job execution time, higher job completion rate, etc.

[0013] In some embodiments, the historical information is updated periodically based on the dynamic nature of the computing environments, including factors such as resource update, peaking time of certain data centers in the computing environments, success rate of submitted jobs, job types, etc. In some embodiment, the historical information may need to contain data that is sufficient for analysis and training, in terms of the amount of data (e.g., thousands of job submission results) or in terms of time (e.g., days, weeks, or even months).

[0014] In some embodiments, the PJAS may include a machine learning (ML) model trained by using reinforcement learning algorithms. A trained PJAS, when submitting (i.e., allocating and distributing) jobs, may be able to look ahead and predict the probabilities of successful completion of the job (also referred to as job success rates) being submitted and the remaining (or unprocessed) jobs to be submitted to make optimal job allocation decisions. For example, when the first of a number of jobs is submitted, that submission may immediately affect the job success rate of the next one or more future jobs because the performance of the computing environment may change after receiving the submitted job. Such a technique (referred to as lookahead) can help achieve optimal performance for the whole computing environments, not just individual jobs.3KILPATRICK TOWNSEND 78819593 1

[0015] In some embodiments, the reinforcement learning (RL) model (e.g., predictive core engine (PCE)) in the PJAS) may trained using rewards to improve its prediction. The rewards may be generated based on the number of successful jobs and aggregated execution time of the jobs. A set of constraints obtained based on analyzing the historical information may be used to help predict the job success rate of each job.

[0016] In some embodiments, the PJAS may use a hybrid approach by combining reinforcement learning and other types of decision-making mechanisms, such as rulebased or heuristic methods. For example, before a sufficient amount of historical information is available for RL-based model to generate good predictions, the combination of two or more decision-making mechanisms can help make optimal decisions in various situations.

[0017] For the purpose of this application, submitting a job may include two steps, which include (1 ) allocating the job to a selected computing environment by a predicted core engine (PCE, e.g., an RL-based model) of the PJAS, and (2) distributing the allocated job, by a job allocator (JA, may also be referred to as a job distributor) of the PJAS, to the selected computing environment. For simplicity, the term submitting a job may be used to cover the above described steps, allocating and distributing. Allocating a job may include an allocation decision made by the PCE to reserve (or select) a computing environment (e.g., one or more data centers) for executing the job. Distributing a job may include sending out the job by the JA and received by the reserved / selected computing environment that may execute the received job. In some embodiments, a job that has been allocated (e.g., an allocation decision has been made) may be distributed immediately. In other embodiments, several jobs may be allocated and then distributed sequentially or simultaneously.

[0018] For the purpose of this application, factors (e.g., peak time, job type, etc.) affecting the performance of the job execution in the computing environments may also be referred to as state variables (or attributes or parameters) in the context of reinforcement learning (RL). These factors, as part of the historical information, may be used as constraints (e.g., how jobs should be allocated in light of historical information to achieve higher job success rates, such as submitting a job to a particular computing4KILPATRICK TOWNSEND 78819593 1environment at a particular time or time window) to help determine the number (or counts) of successful jobs (i.e. , jobs submitted with a high probability of completing execution) as part of the reward calculation for RL. The historical information may include, but not limited to, these factors, job status of the submitted jobs, and resource information of the computing environments.

[0019] FIG. 1 is a block diagram illustrating example computing environments 100 for computing job allocation, according to certain embodiments.

[0020] As shown in FIG. 1 , the example computing environments 100 may include three data centers, DC1 110, DC2 112, and DC3 114. A scheduler 120 may be configured to submit multiple jobs (e.g., job1 140, job2 142, job3 144, and job4 146) through a job queue 130.

[0021] These multiple jobs may be received by the scheduler 120 in a sequence of job1 , job2, job3, and job4 at different times, which will be executed by one or more of the data centers. The scheduler 120 may put these jobs in a temporary queue 130, which may be part of the scheduler 120 or separate from the scheduler 120, to act as a buffer to handle the difference between the receiving rate and the outputting rate. The jobs may be submitted sequentially or roughly simultaneously to the data centers in the computing environments 100.

[0022] In some embodiments, a computing environment may include one or more data centers (also referred to as data systems) that may be connected to each other. Each data center may include a number of individual computers. Different computing environments may or may not be isolated from each other depending on security considerations and implementations. Data centers in a computing environment may be located in the same or different geographic areas. For example, in certain embodiments, DC1 110 may be located on the West coast of the U.S., and DC2 112 may be located on the East coast of the U.S. In other embodiments, these data centers may be located on different continents; for example, DC1 110 is located in the U.S., and DC3 114 is located in Europe. As a result, the peak execution time for each data center may be different due to different time zones. Additionally, because of the distance, time, and location, the5KILPATRICK TOWNSEND 78819593 1performance and network bandwidth for each data center may fluctuate throughout the day.

[0023] In some embodiments, resource available in each data center may be different. Some data centers may be more powerful, with faster processors and higher memory capacity than others, and as a result, different data centers may be suitable for different job types. For example, a type of job called spark job, which is a big data processing job, may be executed in certain data centers. Otherwise, its failure rate may be high. Other types of jobs may include, but not limited to, Java JAR file, Python scripts, MySQL, Hadoop jobs, etc.

[0024] In FIG. 1 , the scheduler 120 may be able to submit the received jobs to data centers in one or more computing environments. For example, job 1 140 may be submitted to DC3 114 at time t1 , which is successfully completed. Job2 142 may be submitted to DC1 110 at time t2, which is successfully completed. Job 144 may be submitted to DC2 112 at time t3, which is also successfully completed. However, job4 146 may be submitted to DC1 110 at time t4, but it fails to complete. After learning the job failure of job4 136, scheduler 120 may re-submit job4 146 to a different data center, such as DC3 114, at time t5, and job4 146 is eventually completed.

[0025] Several factors may contribute to the job failure of job4 146, although both job1 140 and job4 146 are submitted to DC1 110. For example, as mentioned above, the job submission time t4 may coincide with the peak cycle of DC1 110, making job4 146 more likely to fail or take longer time to complete. As another example, job4 146 may be a different type of job from job1 140, and is more suitable for running on DC3 114 rather than DC1 110. The above example may illustrate the dynamic nature of the computing environments. Without knowledge of the computing environments and keeping up-to-date with their dynamic nature, a scheduler may need to perform several retries to submit one or more jobs. Thus, a better approach may be desirable.

[0026] FIG. 2 is a block diagram illustrating an example architecture 200 of a proactive job allocation system (PJAS), according to certain embodiments. Architecture 200, depicted in FIG. 2, is an example and is not intended to unduly limit the scope of claimed embodiments. Many variations, alternatives, and modifications are possible. For6KILPATRICK TOWNSEND 78819593 1example, in some implementations, architecture 200 may have more or fewer components than those shown in FIG. 2, may combine two or more components, or may have a different configuration or arrangement of components. The system, subsystems, and other components depicted in FIG. 2 may be implemented in software (e.g., code, instructions, program) executed by one or more processing units (e.g., processors, cores) of the system, using hardware, or combinations thereof. The software may be stored on a non-transitory storage medium (e.g., on a memory device).

[0027] In FIG. 2, a PJAS 210 is placed between a scheduler 220 and computing environments 240 containing multiple data centers, DC1 250, DC2 252, DC3 254, and DC4 256. Scheduler 220 may forward the jobs to be submitted to PJAS 210, which can submit them on behalf of scheduler 220. PJAS 210 may include, but not limited to, a predictive core engine 230, a job allocator 232, and a control unit 234. A database 236 may be used to store historical information related to submitted jobs. In some embodiments, the database 236 may be integrated with the PJAS 210.

[0028] As shown in FIG. 2, the predictive core engine (PCE) 230 may include a trained reinforcement learning (RL) model responsible for computing the probabilities of successful completion of the jobs (i.e., job success rates) in different computing environments, and selecting the optimal computing environments (e.g., one or more data centers) to submit the jobs. This selection process may be referred to as a job allocation decision. The PCE 230 may consider several computing environmental factors (also referred to as state variables for RL training environment (described below)), such as historical success rates of jobs, job submission timestamps, job execution time, job types, hardware resources of the computing environments. These factors may be part of the historical information that the PCE 230 analyzes for computing the job success rates. The historical information may also include job status, and resource information of the computing environments.

[0029] The job allocator (JA) 232 may take the selection (i.e., allocation decision) from the PCE 230, and distribute the jobs to the selected data centers. In addition to the job distribution, the job allocator 232 may also monitor the status of each job, keeping track of whether a job is running, completed, or has failed. If a submitted job fails, the PCE 2307KILPATRICK TOWNSEND 78819593 1and the job allocator 232 can retry and resubmit the job based on the new selection or recommendation from the PCE 230. Such information of job successes and failures can be collected by JA 232, and provided as feedback to the PCE 230 to improve its future decisions.

[0030] The control unit 234 may supervise or monitor the operations of both the PCE 230 and the JA 232, ensuring they function and work together properly. The control unit 234 may inform system operators to intervene if some problems (e.g., adding, updating, or removing state variables) occur.

[0031] As mentioned above, the PCE 230 may be an RL-based model. In some embodiments, PJAS 210 may utilize a hybrid approach by combining reinforcement learning and other types of decision-making mechanisms, such as rule-based or heuristic methods. The control unit 234 may be responsible for integrating different mechanisms to be used by PJAS 210. For example, a rule-based decision-making method or mechanism may include certain rules to be followed when allocating / submitting jobs. As an example, certain jobs may be tagged with priority levels (i.e., assigning priorities to jobs) indicating that they should be allocated using a rule-based mechanism. Otherwise, the default mechanism is using the reinforcement learning algorithm. As another example, if the default reinforcement learning (e.g., PCE 230) provides selections for a particular job, causing it to fail more than a threshold number of times (e.g., five to ten times), the control unit 234 may temporarily switch the decision-making mechanism to the rule-based mechanism until a later time when PCE 230 improves its predictions (i.e., decisionmaking for job allocations). Such a situation may occur during the initial iterations of predictions from PCE 230 because limited historical information is available.

[0032] In some embodiments, a heuristic method may assign higher priorities for allocating jobs to one or more computing environments that have available resources (e.g., CPUs or memory) or low workloads (e.g., few running jobs). In other embodiments, all three methods (or mechanisms), the reinforcement learning technique, rule-based method, and heuristic method, may be combined and switched between them depending on the received jobs and the computing environments.8KILPATRICK TOWNSEND 78819593 1

[0033] FIG. 3 illustrates an example flowchart depicting processing performed by a proactive job allocation system (PJAS), according to certain embodiments. The method presented in FIG. 3 and described below is intended to be illustrative and non-limiting. Although FIG. 3 depicts the various processing steps occurring in a particular sequence or order, this is not intended to be limiting. In certain alternative embodiments, the processing may be performed in some different order or some steps may also be performed in parallel. It should be appreciated that in alternative embodiments the processing depicted in FIG. 3 may include a greater number or a lesser number of steps than those depicted in FIG. 3.

[0034] At step 310, a plurality of jobs may be received, by a proactive job allocation system (PJAS) comprising a predicative core engine and a job allocator, from a scheduler. For example, in FIG. 2, a scheduler 220 that is responsible for submitting a plurality of jobs to computing environments 240 may forward the job to PJAS 210, which includes a PCE 230, a job allocator 232, and a control unit 234.

[0035] At step 320, probabilities of successful completion of the plurality of jobs with respect to a plurality of data systems may be predicted by the PCE based on historical information related to the plurality of jobs. For example, in FIG. 2, in some embodiments, PCE 230 may use a technique called lookahead (discussed below in relation to FIGs. 5 and 6) by predicting the job success rates of many jobs to be allocated as a whole, and also the job success rate of each individual job when allocating jobs in sequence. The reason is that each submitted job to a computing environment may affect the job success rates of the remaining jobs. Thus, such practice can help achieve optimal allocation decisions (i.e. , prediction) for the jobs as a whole. The allocation decisions may be made by analyzing the historical information that is periodically updated.

[0036] At step 330, job allocation decisions for allocating the plurality of jobs to the plurality of data systems may be generated by the predictive core engine based on the predicted probabilities of successful completion of the plurality of jobs. The PCE 230 may explore and analyze a few scenarios of job allocations for a number of jobs based on their predicted job success rates individually and as a whole to make a job allocation decision for each job. For example, in FIG. 2, considering a scenario that jobs 1 , 2, and 3 allocated9KILPATRICK TOWNSEND 78819593 1to DC1 250, DC2 252, and DC3 254 (denoted as J1 -> DC1 , J2 -> DC2, and J3 -> DC3) in such sequence may have high job success rates for job 1 and job 2, but not for job 3. Thus, the overall probability of job success rate (e.g., number of successful jobs) may be lower than a different scenario (e.g., J1 -> DC2, J2 -> DC3, and J3 -> DC4). Both scenarios may be analyzed by PCE 230 before making an allocation decision for each job.

[0037] In some embodiments, a hybrid approach, such as combining RL and rule-based method, may be used by PCE 230 to make allocation decisions, for example, when retry thresholds are reached for one or more jobs.

[0038] At step 340, the plurality of jobs may be distributed by the job allocator to the plurality of data systems based on the allocation decisions. Continuing with the above example, if PCE 230 decides to use scenario 2 (e.g., J1 -> DC2, J2 -> DC3, and J3 -> DC4), the JA may distribute J1 to DC2, then J2 to DC3, and finally J3 to DC4. JA may monitor the job status for all jobs, and update their status to the historical information stored in database 236, accordingly. In some embodiments, the J1 status may be received and updated to the historical information before job J3’s allocation. In such a situation, the PCE 230 may be able to utilize that updated information when making allocation decisions for J3, for example, allocating J3 to DC2 rather than DC4 because DC2 yields high-performance results for J1 .

[0039] FIG. 4 is a simplified block diagram of an RL training environment 400 for training a predictive core engine (PCE) within PJAS, according to certain embodiments. The training environment 400 depicted in FIG. 4 is merely an example and is not intended to unduly limit the scope of claimed embodiments. Many variations, alternatives, and modifications are possible. For example, in some implementations, training environment 400 may have more or fewer systems or components than those shown in FIG. 4, may combine two or more systems, or may have a different configuration or arrangement of systems.

[0040] As shown in FIG. 4, the training environment 400 may include, but not be limited to, an agent 410 containing the PCE 230, and an environment 420 including the10KILPATRICK TOWNSEND 78819593 1computing environments 240. Reinforcement learning (RL) techniques may be used for training PCE 230.

[0041] Reinforcement learning is a goal-directed computational approach where an agent may learn to make decisions (or perform a task) by interacting with a dynamic environment. Unlike supervised learning, RL enables the agent to make a series of decisions to maximize the cumulative reward for the task without being explicitly programmed to achieve the task. In other words, the agent may learn the optimal behavior through repeated trial-and-error interactions with the environment. RL training environment may include a few components, such as an agent 410, an environment 420, state (440 or 460), action 430, and reward (450 or 470).

[0042] An agent 410 may be a sub-system or a model (e.g., PCE 230) that includes a policy and an RL algorithm (e.g., based on Markov decision process (MDP)), where the model can learn and make decisions. The model running RL algorithm may be referred to as RL-based model. In some embodiments, the PCE 230 can be model-based RL or value-based RL. An environment 420 (e.g., the computing environments 240) may be the problem space where the agent operates. The state may refer to the current situation or configuration of the environment at any given time. The state can be a current state 440 (“St” at time t) received and used by the agent 410 / PCE 230, or the next state 460 (“St+i” at time t+1 ) outputted from the environment 420 after receiving the action 430.

[0043] Action 430 (At at time t) may be a decision or a step taken by the agent within the environment. Reward may refer to the feedback received after the action 430, and can be positive, negative, or neutral. The reward can be a current reward 450 (Rt at time t) received and used by the agent 410 / PCE 230, or the next reward 470 (“Rt+i” at time t+1 ) generated based on the environment 420 after receiving the action 430. In some embodiments, RL may utilize cumulative reward (e.g., the total sum of rewards collected over time), which the agent aims to maximize.

[0044] The policy in the agent 410 / PCE 230 may be a mapping that selects actions based on the current state 440 (also referred to as observation) from the environment. The policy may be a function of approximator with tunable parameters, such as a deep neural network. The RL algorithm may continuously update the policy parameters11KILPATRICK TOWNSEND 78819593 1(referred to as model parameters of the reinforcement learning model (RL model)) based on the action, state, and reward. The goal of the learning algorithm is to find an optimal policy that maximizes the cumulative reward received by the agent 410 / PCE 230 when performing tasks.

[0045] To illustrate the training process, at time t, the agent 410 / PCE 230 may generate an action (At) 430 (e.g., an allocation decision, or allocation of a data centerfor a particular job). The particular job is then distributed, by the job allocator 232, to a computing environment 240 containing this selected data center. At time t+1 , next state 460 (St+i) and next reward 470 (Rt+1 ) are provided to the agent 410 / PCE 230, and become current state 440 (St) and current reward 450 (Rt). The current state 440 and current reward 450 may affect the agent’s action 430 (At+i at time t+1 ), which starts the next iteration.

[0046] Because the agent 410 may consider the current state 440 (St) and policy to generate action 430 (At, e.g., making a decision), which may lead to a next reward 470 (Rt+1 ) that becomes the current reward 450 for training the agent, the reward may be deemed a function that maps state-action pairs to numerical values as rewards. In some embodiments, the reward function may be implemented as a table. Further details describing the reward function are described below in FIG. 5.

[0047] In some embodiments, the variables for the state (440 or 460) of the environment 420 (e.g., the computing environments 420) may include, but not limited to, historical success rates of jobs, job submission timestamps, job execution time, job types, hardware resources of the computing environments, and job submission pattern (or order).

[0048] The historical success rates of jobs can indicate the pattern of success or failure of a job in a particular computing environment, such as a data center. For example, suppose a job J1 has been submitted to data center DC 1 ‘m’ number of times (e.g., 1000 times), and it has successfully completed ‘p’ number of times but failed f number of times, ‘p’ and T together equal ‘m.’ If ‘p’ is significantly higher than ‘f,’ PCE 230 may interpret that job J1 has a high probability of success in data center DC1.

[0049] On the other hand, if ‘p’ is significantly lower than ‘f, PCE 230 may interpret that job J1 has a high probability of failure in data center DC1. Under such situation, PCE 23012KILPATRICK TOWNSEND 78819593 1may prioritize submitting job J1 to other computing environments, such as DC2 ~ DC4. The purpose of this state variable, historical success rates of jobs, can provide information to PCE 230 to increase the success rate of job execution and optimize the use of computing resources.

[0050] Job submission timestamp may track the time (or time window) when a job is submitted to a particular computing environment (e.g., a data center). Because resource utilization may fluctuate in compute environments and scheduled maintenance may cause interruptions or delay, the timing of submitting a job to a particular data center may be important. For example, during peak hours, an environment may be handling a large number of concurrent jobs, each requiring a portion of the available memory resources. The cumulative demand during these peak hours can lead to high memory usage, reducing the resources available for new jobs. This scarcity of resources can result in execution delays, performance degradation, or even job failures.

[0051] Job execution time refers to the estimated duration a job takes to complete. For example, certain computing environments may struggle to efficiently handle jobs with long execution times, leading to a higher likelihood of job failures in these environments because resources may not be consistently available over an extended period of time. Consider a scenario where a job J1 is submitted to data center DC1 , which may have a moderate amount of resources, during off-peak hours. However, the submitted job J1 may have a lot of data to process, and take a long time to complete. If the job execution continues until the peak hours for the data center DC1 , the resources for data center DC1 may be further strained, leading to a higher likelihood of job failure for job J1 .

[0052] Job types may refer to various types of jobs that require different amounts of computing resources (e.g., CPUs, memory, storage, etc.), execution time, and network bandwidth. For example, some types of jobs, such as spark jobs or Hadoop jobs, utilize parallel processing of multiple CPU cores, perform reading and writing to large amounts of data. As a result, such jobs may need more powerful CPUs, consume large amounts of memory, use a lot of storage I / O bandwidth, and move data across network nodes. Thus, PCE 230 may choose to allocate these types of jobs to certain computing environments to increase their success rate.13KILPATRICK TOWNSEND 78819593 1

[0053] Hardware resources of the computing environments may refer to resources, such as processing power (CPU), memory (RAM), storage capacity, and network bandwidth. As discussed above, certain job types may need to use certain computing environments to achieve optimal performance. For example, some jobs may need a computing environment with more computational power, while other jobs involving large amounts of data may need another computing environment with ample storage capacity and memory. Thus, by considering the hardware resources of each computing environment, the PCE 230 can make informed decisions about where to allocate jobs.

[0054] Job submission pattern / order may refer to the jobs received by the PJAS 210 from the scheduler 220 and intended to be submitted in a particular order. The Job submission pattern is a factor for the PCE 230 of the PJAS 210 to consider because when a first job is submitted, that first job can take up resources from the selected computing environment (e.g. , one or more data centers) and thus affect the choices of the remaining jobs to be submitted.

[0055] In some embodiments, these state variables may be interrelated, and changes to one state variable may affect another variable over time. For example, if the hardware resources of a particular computing environment (e.g., one or more data centers) have been upgraded, which changes one state variable (for hardware resources), the job execution time (i.e., another state variable) for the jobs submitted to this computing environment may go down. After a few iterations, PCE 230 may learn from the state changes 440 (including changes to the state variables of job execution time and hardware resources) that certain jobs submitted to the data centers in this particular computing environment (e.g., reflected in the action 430) can lead to more positive rewards 450. The policy parameters (or model parameters of RL-model) may be updated accordingly and adapted to the environmental change (i.e., upgraded hardware resources of this particular computing environment). In other words, the historical information of these state variables may be collected, and may be considered by PCE 230 during its decision-making process.

[0056] FIG. 5 is a table illustrating an example reward function for reinforcement learning techniques, according to certain embodiments. As discussed above in relation14KILPATRICK TOWNSEND 78819593 1to FIG. 4, a reward may be generated based on a state-action pair. In FIG. 5, a reward function is implemented as a mapping table 500 that maps state-action pairs to rewards for reinforcement learning techniques.

[0057] The state column 510 in FIG. 5 may include variables, such as job submission pattern, job submission timestamp, historical success rate of jobs, job types, and computing environments (e.g., data centers) available. The action column 520 indicates which data centers are selected for which jobs to submit. The reward column 530 is the score (e.g., a numerical value) generated based on the state variables received and action output by the PCE 230 (or agent 410). Please note that not all state variables discussed above in FIG. 4 are used to simplify the example. The state variables used for calculating or generating a reward may also be referred to as parameters for reward calculation.

[0058] In some embodiments, the reward function may generate a relative reward score by considering certain counts of successful jobs (i.e., high probability of successful completion of jobs) and overall / aggregated execution time (by aggregating these jobs’ respective execution times), as illustrated in FIG. 5. A relative reward score may mean a reward score for a state-action pair may not be an absolute value, but is relative to reward scores of other state-action pairs. The successful jobs in the counts may be determined based on certain constraints and observations after analyzing the historical information (e.g., state variables, status (success / failure) of submitted jobs, and compute environments).

[0059] The constraints may be created by identifying and analyzing the relation between the state variables (attributes or factors), job status of submitted jobs, and resources of the computing environment to provide certain criteria that can help achieve a high job success rate for a job. In some embodiments, the execution time of an allocated job to a particular computing environment (e.g., one or more data centers) may be estimated based on the number of constraints met or violated and categorized as long, medium, and short. For example, if an allocated job to a particular data center meets all constraints, its execution time is estimated to be low. Each violated constraint may increase its execution time. For example, one violation may be estimated to have a medium execution15KILPATRICK TOWNSEND 78819593 1time, and two or more violations may be estimated to have a long execution time or high failure rate.

[0060] An example of constraints and observations (in the form of statements) for such a purpose is listed below.Definitions used in the below constraints:J refers to job submission pattern. “J1 -> J2 -> J3” means the order of job submission of three jobs (J1 , J2, and J3). t refers to job submission timestamp, the time at which a job is submitted.Type refers to job type for denoting specific kind of jobs.Et refers to job execution time (or the total running time of job).S refers to the historical success rate of a particular job, which may be statistical data of failures and successes.C refers to computing environments, such as data centers. k refers to a positive constant, which may be a scalar used for reward score.Example constraints and statements:1 . t3 is a peak cycle for C1 and C2. (This constraint means that submitting J3 to C1 or C2 at timestamp t3 may have a high failure probability due to their peak cycles.)2. S1 C2 « S1 C1 and S1 C1 == S1 C3. (This constraint means that J1 has a significantly higher historical success rate (S) on compute environment C1 (denoted as S1 C1 ) than on C2. J1 running on C1 has roughly equal historical success rate as J1 running on C3 (denoted as S1 C3).3. J3 is a spark job that has a higher success rate on C4 and C1 than in C3. (This constraint means J3 is a particular type of job, called spark job, that requires more16KILPATRICK TOWNSEND 78819593 1powerful computing environments with more memory. J3 may have a higher success rate when running on compute environments C1 and C4, but a lower success rate when running on C3.)4. S2C1 == S2C2 == S2C3 == S2C4 . (This statement means that job J2 may have equivalent success rates in all the computing environments, C1 , C2, C3, and C4.)

[0061] Referring to table 500 in FIG. 5, four cases or scenarios are illustrated. A case or scenario may refer to a state-action pair and its corresponding reward score. In table 500, jobs are submitted in the sequence of J1 , J2, and J3 (denoted as J1 -> J2 -> J3) at timestamps t1 , t2, and t3, respectively. The constraint / statement #1 is included in the second column for the variable of timestamp. The constraint / statement #2 is included in the third column for the variable of historical success rates. The constraint / statement #3 is included in both the fourth and fifth columns for the variable of job type and computing environments.

[0062] In the first case (case #1 ), the action column has J1 -> C2 (J1 is submitted to C2), J2 -> C4 (J2 is submitted to C4), and J3 -> C1 (J3 is submitted to C1 ). Based on the above constraints and statements, this first case may have only one count of successful jobs, resulting in a reward score of 0.2k, as explained below (based on constraints / statements and probability of successful completion of each job (or job success rate):J1 -> C2 : low probability of successful completion based on constraint / statement #2J2 -> C4 high probability of successful completion based on constraint / statement #4J3 -> C1 : low probability of successful completion based on constraint / statement #117KILPATRICK TOWNSEND 78819593 1

[0063] In the second case (case #2), the action column has J1 -> C3, J2 -> C4, and J3 -> C2. Based on the above constraints and statements, this second case may have two counts of successful jobs, resulting in a reward score of 0.4k, as explained below:J1 -> C3 high probability of successful completion based on constraint / statement #2J2 -> C4 high probability of successful completion based on constraint / statement #4J3 -> C2 : low probability of successful completion based on constraint / statement #1 and #3

[0064] In the third case (case #3), the action column has J1 -> C1 , J2 -> C2, and J3 -> C3. Based on the above constraints and statements, this third case may have two counts of successful jobs, resulting in a reward score of 0.6k, as explained below:J1 -> C1 high probability of successful completion based on constraint / statement #2J2 -> C2 high probability of successful completion based on constraint / statement #4J3 -> C3 : low probability of successful completion based on constraint / statement #3

[0065] Although both the second case and the third case have two counts of successful jobs, J3 in the third case may have a shorter execution time than in the second case because J3 in the second case violates two constraints (#1 and #3) while J3 in the third case violates one constraint (#3). Thus, the third case has a higher reward score (0.6k) than the second case.

[0066] In the fourth case (case #4), the action column has J1 -> C1 , J2 -> C2, and J3 - > C4. Based on the above constraints and statements, this fourth case may have three counts of successful jobs, resulting in a reward score of 0.9k, as explained below:18KILPATRICK TOWNSEND 78819593 1J1 -> C1 high probability of successful completion based on constraint / statement #2J2 -> C2 high probability of successful completion based on constraint / statement #4J3 -> C4 high probability of successful completion based on constraint / statement #3

[0067] Because the fourth case has the highest counts (e.g. , three counts) of successful jobs and the shorter execution time for all submitted jobs (e.g., meeting all constraints and using preferred computing environments), it generates the highest reward score (0.9k). On the other hand, The first case has the lowest count (e.g., one count) of successful jobs because two submitted jobs have a high failure probability that is likely to turn into a long execution time. Thus, it generates the lowest reward score (0.2k).

[0068] In some embodiments, weights may be applied to the state variables (or parameters) used in calculating or generating the reward to indicate the importance of each variable / parameter. For example, when a new variable is introduced, a lower weight may be applied to this new variable. Over time, the RL training environment 400 may update and adjust the weight of this new variable or other variables to make better decisions. When a variable is not used, a zero weight may be applied to that variable. The weights may be applied dynamically after one or more iterations of the RL training.

[0069] In some embodiments, jobs may be received and submitted in sequence, but the agent 410 (or PCE 230) may look ahead (also called lookahead technique) and predict the probabilities of job success rate for not only the job being considered but also all remaining jobs not yet received or to be submitted (i.e., the future job in the submission pipeline) because agent 410 / PCE 230 may try to maximize the cumulative reward. For example, when job J1 is submitted, that submission may immediately affect the job success rates of the next one or more jobs because the performance of the computing environments may change after receiving the submitted job J1 . In other words, agent 410 / PCE 230 may try to make a decision that is optimal for the overall performance of the19KILPATRICK TOWNSEND 78819593 1computing environments based on all potential jobs in the submission pipeline, not just making a decision for one job at a time in isolation.

[0070] To illustrate, suppose that at time t1 , J1 is allocated to (or to be submitted to) C2, as shown in case 1 in Table 500. Based on prior received rewards, the PCE 230 may be aware that the other potential jobs in the submission pipeline or remaining jobs, J2 and J3, should not be submitted to C4 and C1 , respectively, at times t2 and t3, because such selections may lead to low reward score based on prior experience in case 1 . Thus, the PCE 230 may explore submitting J2 to C3 instead, in which such selection has not been done before. Based on that exploration, J3 can then have two options of being submitted to C1 or C4. If PCE 230 submits J3 to C1 , it may get a low reward score. If PCE 230 submits J3 to C4, it may get a higher reward score. Such explorations by PCE 230 during RL training may help discover new strategies but also have a risk of getting low rewards. The explorations by agent 410 / PCE 230 are explained below:Exploration 1 (or case 5):At time t1 , J1 -> C2 : low probability of successful completion based on constraint / statement #2At time t2, J2 -> C3 : high probability of successful completion based on constraint / statement #4At time t3, J3 -> C1 : low probability of successful completion based on constraint / statement #1Exploration 2 (or case 6):At time t1 , J1 -> C2 : low probability of successful completion based on constraint / statement #2At time t2, J2 -> C3 : high probability of successful completion based on constraint / statement #4At time t3, J3 -> C4 : high probability of successful completion based on constraint / statement #320KILPATRICK TOWNSEND 78819593 1

[0071] Both case 5 and case 6 have not been explored by PCE 230 before. After trying case 5, the reward score may be 0.2k because it has only one count of successful jobs. However, case 6 may have a better reward score of 0.6k because it has two counts of successful jobs. In other words, agent 410 / PCE 230 may be able to estimate rewards for its planned actions or decisions (e.g., submitting J2 and J3 in the future) when it has a lot of historical information to analyze (e.g., many cases with reward scores). This information (cases 5 and 6) may eventually become part of the historical information that PCE 230 can analyze and evaluate during its lookahead process.

[0072] To summarize an example of the lookahead technique with the above analysis and explorations, suppose PJAS receives three jobs, J1 , J2, and J3. Agent 410 / PCE 230 in PJAS may analyze the jobs individually and as a whole. When agent 410 / PCE 230 analyzes J1 to make an allocation decision, it may use data related to J1 and other unprocessed jobs (e.g., J2 and J3) waiting to be processed for such analysis. When analyzing J2 to make a second allocation decision, agent 410 / PCE 230 may use data related to J2 and the remaining or unprocessed job (e.g., J3) waiting to be processed for such analysis. The data used for the analysis (e.g., predicting job success rates) may include constraints, which may involve attributes used for RL, the jobs being analyzed, and the available computing environments for allocation and distribution. As an example, agent 410 / PCE 230 may allocate J1 to C2, while predicting job success rates for all three jobs. When allocating J2, agent 410 / PCE 230 may perform the same analysis or explorations again for J2 and J3 (i.e., the remaining / unprocessed jobs for allocation) with updated historical information (i.e., updated data). The same analysis may lead to the decision to allocate J2 to C3. Finally, for J3 allocation, the same analysis may be performed for J3 only (i.e., the only remaining job for allocation), by checking the constraints (i.e., part of the data used for analysis) and the updated historical information.

[0073] To further illustrate, based on prior experience (e.g., historical information), agent 410 / PCE 230 may be aware that submitting J1 to C1 tends to have a higher reward score, as shown in case 3 and case 4 of Table 500. For other potential jobs (e.g., J2 and J3) in the submission pipeline, they can be submitted to either C1 , C3, or C4. Agent 410 / PCE 230 may explore these options to estimate reward scores before submitting J2, then J3.21KILPATRICK TOWNSEND 78819593 1If PCE 230 repeats the prior actions, it may maximize immediate rewards but miss discovering better strategies. Such an approach may be referred to as exploitation as compared to the exploration discussed above.

[0074] In some embodiments, the RL training environment 400 may balance the exploration and exploitation by utilizing various ways and techniques, including but not limited to, Epsilon-greedy strategy, Upper Confidence Bound (UCB), softmax action selection, etc.

[0075] FIG. 6 is an example flowchart illustrating a method (e.g., RL) for training a predictive core engine (PCE) within PJAS, according to certain embodiments. The following steps, 610 ~ 680, may be repeated for each training iteration.

[0076] At step 610, historical information that includes two or more attributes and is related to a plurality of data systems may be obtained. For example, in FIG. 2, historical information stored in database 236, including attributes that can be used for training PCE 230 using RL techniques, may be obtained. The attributes may include, but are not limited to, historical success rates of jobs, job submission timestamps, job execution times, job types, and hardware resources of the data systems. Those attributes, when applied to the submitted jobs, may become constraints to give the RL training environment 400 to determine whether the allocation decisions (i.e. , action 430 of FIG. 4) are on the right track in the form of rewards, as shown in Table 500 of FIG. 5. The attributes together with the constraints may be equivalent to the current state, S(t), 440.

[0077] At step 620, one or more jobs for allocation may be received. For example, in FIG. 5, jobs J1 , J2, and J3, may be received and ready to be allocated by the agent 410 / PCE 230 of FIG. 4.

[0078] At step 630, a predicted probability of the successful completion (i.e., job success rate) of each of the one or more jobs may be generated by a predictive core engine based on the historical information. For example, in FIG. 4, the agent 410 / PCE 230 may predict the job success rate of each job in light of the constraints as described in relation to FIG. 5 by evaluating different historical scenarios of job allocation decisions for jobs J1 , J2, and J3. As an illustration, agent 410 / PCE 230 may find that allocating J1 to C1 may meet22KILPATRICK TOWNSEND 78819593 1the constraints (i.e. , predicting potentially high job success rate) and also lead to higher rewards in the past, as shown in Table 500 of FIG. 5. Allocating J2 to C4 may also meet the constraints but lead to lower rewards in the past, as shown in Table 500 of FIG. 5.

[0079] At step 640, an action may be generated based on the evaluation of the predicted probabilities of the successful completion the one or more jobs. For example, an action (A(t)) 430 of FIG. 4, which is the allocation decisions of the one or more jobs (e.g., J1 , J2, and J3), may be made, as shown in the action column 520 in Table 500 of FIG. 5. Continuing with the above example, based on the above evaluation / observation of the historical information, agent 410 / PCE 230 may generate an allocation decision: J1 -> C1 , J2 -> C2, and J3 -> C3, as shown in case #3 of Table 500 of FIG. 5.

[0080] At step 650, a reward may be determined based on the predicted probability of the one or more jobs and the attributes. As discussed above, a relative reward score may be generated based on counts of successful jobs (i.e., high probability of successful completion of jobs) and their respective execution times. For case #3, the counts of successful jobs may be two out of three based on the constraints discussed in relation to FIG. 5. The total execution time for jobs J1 -J3 may be moderate because only one job, J3, violates constraint #3. The determined reward may be equivalent to or referred to as the next reward R(t+1 ) 470 of FIG. 4, which is calculated based on state S(t) 440 and Action A(t) 430.

[0081] At step 660, model parameters in the predicative core engine (PCE) may be updated based on the determined reward. The model parameters (e.g., weights of neural network) of the RL model (e.g., PCE) may be updated based on the determined reward to maximize the cumulative reward. The reward that is used to update the model parameters of agent 410 / PCE 230 may be equivalent to or referred to as the current reward R(t) 450 of FIG. 4.

[0082] At step 670, job status of the one or more jobs and any updates to the attributes may be collected. After a job has been distributed to the selected computing environment (e.g., one or more data centers), job allocator (JA) 232 may monitor the status of the job, such as successful completion or failure, and collect any updates to the attributes, such as job execution time, change of hardware resources, job success rate (e.g., # of23KILPATRICK TOWNSEND 78819593 1success / # of submission of a particular job), etc. The status and updates to the attributes may be equivalent to or referred to as next state S(t+ 1 ) 460 of FIG. 4.

[0083] At step 680, the historical information may be updated based on the collected job status and updated attributes. The job allocator 232 may then update the historical information stored in the database 236 with the new status of the one or more submitted jobs and attributes. This updated information can be used by the agent 410 / PCE 230 for the next iteration of RL training, and is equivalent to or referred to as current state S(t) 440 of FIG. 4. In other words, the next state S(t+1 ) 460 generated from the environment 420 becomes current state S(t) 440 when it is consumed or used by the agent 410 / PCE 230.

[0084] The above description is illustrative and is not restrictive. Many variations of the invention will become apparent to those skilled in the art upon review of the disclosure. The scope of the invention should, therefore, be determined not with reference to the above description, but instead should be determined with reference to the pending claims along with their full scope or equivalents.

[0085] One or more features from any embodiment may be combined with one or more features of any other embodiment without departing from the scope of the invention.

[0086] As used herein, the use of "a," "an," or "the" is intended to mean "at least one," unless specifically indicated to the contrary.24KILPATRICK TOWNSEND 78819593 1

Claims

WHAT IS CLAIMED IS:1 . A method, comprising: receiving, by a proactive job allocation system (PJAS) comprising a predictive core engine and a job allocator, from a scheduler, a plurality of jobs; predicting, by the predictive core engine, probabilities of successful completion of the plurality of jobs with respect to a plurality of data systems based on historical information related to the plurality of jobs; generating allocation decisions, by the predictive core engine, for allocating the plurality of jobs to the plurality of data systems based on the predicted probabilities of successful completion of the plurality of jobs; and distributing, by the job allocator, the plurality of jobs to the plurality of data systems based on the allocation decisions.

2. The method of claim 1 , wherein the predictive core engine generates the allocation decisions by individually analyzing the jobs in the plurality of jobs.

3. The method of claim 2, wherein each analyzed job uses data relating to unprocessed jobs waiting to be processed in an allocation decision for the analyzed job.

4. The method of claim 3, wherein the data relating to the unprocessed jobs comprises attributes of the unprocessed jobs.

5. The method of claim 1 , wherein the predictive core engine uses a reinforcement learning model to predict the probabilities of successful completion of the plurality of jobs.

6. The method of claim 5, wherein the reinforcement learning model is trained using rewards, and wherein the rewards are generated based on the probabilities of successful completion of the plurality of jobs, and aggregated execution time of the plurality of jobs.25KILPATRICK TOWNSEND 78819593 17. The method of claim 1 , wherein the predictive core engine uses a combination of a reinforcement learning model and a rule-based method to predict the probabilities of successful completion of the plurality of jobs.

8. The method of claim 7, wherein the rule-based method comprises assigning priorities to the plurality of jobs.

9. The method of claim 7, further comprising switching from the reinforcement learning model to the rule-based method for a particular job of the plurality of jobs when number of job failures for the particular job exceeds a threshold.

10. The method of claim 1 , wherein the predictive core engine is trained using two or more attributes, the two or more attributes comprising: a) historical success rates of jobs, b) job submission timestamps, c) job execution times, d) job types, and e) hardware resources of the data systems.11 . The method of claim 10, wherein a probability of successful completion of a particular job of the plurality of jobs is determined based on a set of constraints.

12. The method of claim 11 , wherein the set of constraints is generated by identifying relation between the two or more attributes, the particular job of the plurality of jobs, and the plurality of data systems.

13. The method of claim 1 , wherein the historical information comprises factors affecting performance of job execution by the plurality of data systems, job status, and resources of the plurality of data systems.26KILPATRICK TOWNSEND 78819593 114. The method of claim 1 , wherein the PJAS comprises a control unit which controls the predictive core engine and the job allocator.

15. The method of claim 1 , wherein the data systems are data centers or individual computers.

16. A system, comprising: a predictive core engine; a job allocator; and a control unit; wherein the predictive core engine is configured to perform operations comprising: predicting probabilities of successful completion of a plurality of jobs with respect to a plurality of data systems based on historical information related to the plurality of jobs; and generating allocation decisions for allocating the plurality of jobs to the plurality of data systems based on the predicted probabilities of successful completion of the plurality of jobs; and wherein the job allocator is configured to distribute the plurality of jobs to the plurality of data systems based on the allocation decisions.

17. The system of claim 16, wherein the predictive core engine uses a reinforcement learning model to predict the probabilities of successful completion of the plurality of jobs.

18. The system of claim 16, wherein the predictive core engine is trained using two or more attributes, the two or more attributes comprising: a) historical success rates of jobs, b) job submission timestamps, c) job execution times, d) job types, and27KILPATRICK TOWNSEND 78819593 1e) hardware resources of the data systems.

19. The system of claim 16, wherein the historical information comprises factors affecting performance of job execution by the plurality of data systems, job status, and resources of the plurality of data systems.

20. The system of claim 16, wherein the control unit controls the predictive core engine and the job allocator.28KILPATRICK TOWNSEND 78819593 1

Images