[0057] Example one
[0058] This embodiment provides a job scheduling method, such as figure 1 As shown, including the following steps:
[0059] Step 101: For the first job in the waiting queue in the online mode, according to the task information of all jobs before the first job, estimate the first time point when the available resources meet the requirements of the first job;
[0060] Step 102: Compare the first time point with the upper limit of the waiting time of the first job;
[0061] Step 103: When the first time point is greater than the upper limit of the waiting time of the first job, switch the first job to run in an offline mode.
[0062] In this embodiment, when the scheduled job is in the waiting state, the first time point when the available resources can meet the needs of the scheduled job can be estimated at any time according to the situation of the scheduled job and the current running conditions of other jobs. If the first time is greater than the upper limit of the waiting time of the first job, the first job will be directly transferred to the offline mode to run. In this way, it can be placed in the waiting queue according to the running status of the scheduled job itself and the current running status of other jobs The reasonable scheduling of each job improves the processing efficiency and resource utilization in the distributed system.
[0063] In practical applications, a distributed system will run on a cluster composed of multiple physical nodes. The method of this embodiment is implemented by the master physical node (Master) in the cluster. If the main physical node in the cluster structure is implemented by multiple designated physical nodes (Clients) executing a certain algorithm in parallel, then the method of this embodiment can be executed in parallel through the multiple designated nodes.
[0064] In actual applications, after the job is submitted, if the resources are sufficient, it will be scheduled immediately without waiting. If the current available resources do not satisfy the request of the job, the job will be added to the waiting queue in the online mode. In this embodiment, for the first job in the waiting queue in the online mode, the first time when the available resources meet the requirements of the first job can be estimated based on the task information of the unfinished tasks in all the jobs before the first job point. Wherein, the task information includes at least: the running time of the unfinished task, the number of instances, and the running status of each instance.
[0065] In a distributed system, a job contains multiple tasks, and each task contains multiple instances. The instance is the smallest unit of the job, and each instance corresponds to an actual physical process. There is a certain correspondence between instances and resources. Therefore, as many resources as there are instances in a job. The running status of each job in the distributed system can be queried through the running log. The running log contains task information of each task in the job. Each piece of task information corresponds to a task, which can include task identification (such as task ID), job Identification (such as job ID), task running status (such as information identifying task running, information identifying task running end, etc.), task running time (such as running start time, running end time, etc.), number of instances, and each The running status of each instance (for example, information identifying the running of the instance, information identifying the end of the running of the instance, etc.).
[0066] In some implementation manners, the first time point may be estimated in the following manner: according to task information of unfinished tasks in all jobs before the first job in online mode, the task end time of the unfinished task is estimated And the corresponding amount of releasable resources; according to the end time of the task and the corresponding amount of releasable resources, and the current amount of available resources, determine the first moment when the available resources meet the requirements of the first job. In this way, by establishing the mapping relationship between the task end time of each task and the number of releasable resources, the number of available resources at the end of each task can be obtained in real time, and the first time point at which the available resources can meet the requirements of the first task can be determined in real time. .
[0067] Specifically, the task end time of the unfinished task can be obtained according to the running time and the current time point of the unfinished task; the unfinished task can be obtained according to the number of instances of the unfinished task and the running status of each instance The number of unfinished running instances in the unfinished task is determined, and the number of releasable resources of the unfinished task is determined based on the number of unfinished running instances in the unfinished task. In this embodiment, the number of resources that can be released after the task is completed is estimated by the number of instances in the task, which is more accurate.
[0068] In this embodiment, after estimating the task end time and the number of releasable resources of the unfinished tasks in each job, the task end time and the number of releasable resources of the unfinished tasks in each job can be correspondingly saved so that it can be passed Querying the stored information to estimate the first moment is beneficial to improve processing efficiency.
[0069] In this embodiment, the task end time and the number of releasable resources of the unfinished task can be stored in multiple ways. For example, a predefined data structure can be used to save the task ending time and the amount of released resources of the unfinished task. The data structure may include at least the following parts: the first part that records the end time of the task; the second part that records the number of releasable resources; the third part that records the task identification (such as the task ID and the job ID), and so on. For another example, the task ending time and the amount of released resources of the unfinished task can be recorded in the form of a histogram. Of course, other forms besides the above-mentioned forms may also be used in practical applications to store information such as the end time of the task and the corresponding amount of releasable resources.
[0070] In some real-time methods, after the task end time and the number of releasable resources are saved, they can also be updated in real time according to the running status of each task. When a task is finished running, delete the task end time and the number of free resources corresponding to the task. On the one hand, it can ensure that the saved data is most in line with the current operation of the job, which is beneficial to improve the accuracy of the first moment of estimation ; On the other hand, delete unnecessary data in time to save storage space.
[0071] In this embodiment, the upper limit of the waiting time of the first job is determined according to the upper limit of the waiting time of the first job. For example, the upper limit of the waiting time of the first job can be obtained by the sum of the submission time of the first job and the upper limit of the waiting time.
[0072] Among them, the upper limit of the waiting time of the first job can be obtained in various ways. For example, the upper limit of the waiting time may be set by the user, and set in the submission request of the first job shown before the first job is submitted. For another example, the online mode running time and the offline mode running time of the first job can be determined according to the historical running data of the first job, and then the waiting time of the job can be determined according to the online mode running time and the offline mode running time of the first job Maximum duration.
[0073] In some implementations, the historical operation data may include offline mode operation data and online mode operation data, and the waiting time of the first job may be determined based on the offline mode operation data and online mode operation data of the first job Upper limit.
[0074] Wherein, the offline mode operation data of the first job may include: offline mode operation time information of each task in the first job; the online mode operation data of the first job may include: each task in the first job Information about the running time of the task in online mode.
[0075] Specifically, the offline mode running time and the online mode running time of the first job can be determined according to the offline mode running time information and online mode running time information of each task in the first job; The submission time, the offline mode operating time and the online mode operating time determine the upper limit of the waiting time of the first job.
[0076] For example, the sum of the difference between the submission time and the running time in offline mode minus the running time in online mode can be used as the upper limit of the waiting time of the first job. In this way, it is determined that the actual waiting time of the first job is greater than the running time in offline mode. When the difference between the running time in online mode is the first job is transferred to offline mode.
[0077] For another example, a waiting time threshold may be preset, and the maximum value between the waiting time threshold and the difference between the offline mode operating time minus the online mode operating time may be determined, and the sum of the maximum value and the submission time of the first job is taken as the value. The upper limit of the waiting time of the first job is described. Of course, other methods can also be used to determine the upper limit of the waiting time of the first job in practical applications, and this article does not limit this.
[0078] In practical applications, the distributed system runs the same job logic, only the amount of input data is different. When the job is executed according to the input data, the online mode operation data and the offline mode operation data of the job are recorded in the database. In this embodiment, the running time length of a job in the offline mode and the running time length in the online mode can be obtained according to the running data recorded in the database.
[0079] In some implementation manners, the online mode running time and the offline mode running time of the job are estimated based on the historical running data of the job and the current input data volume. That is to say, the running time in the online mode and the running time in the offline mode can be estimated according to the historical running data and the amount of input data of the job. For example, the running time of each task in the job can be estimated based on historical running data, and then the running time of the job can be obtained from the sum of the running time of each task.
[0080] In practical applications, the historical running data of the job may include: the amount of input data of the job, the number of tasks, the number of instances in each task, and the running time of each task. For example, the running time of a single instance of the job can be estimated based on the running time of each task in the job. Since the amount of input data of a job is proportional to the number of instances, the total number of instances of the job in this round can be estimated based on the amount of input data of the job in this round, which can be estimated by the product of the total number of instances and the running time of the single instance The duration of the operation described in this round. Both online mode operation time and offline mode operation time can be estimated by the above method. In addition, other methods can also be used for estimation, and this article is not limited. For a running job, you can get the task information of the running job from the running log, including information about which tasks have ended and which tasks are running. For a running task, you can also get information about how long the task needs to run (that is, the running time), which instances of the task are running, and so on. Based on this information, you can get the task information of the job and estimate its running task information. , The running task information may at least include the task ending time of the running task, the number of releasable resources, task ID, job ID, etc. Among them, the task ending time of the running task can be obtained from the current time and the running duration of the task, and the number of releasable resources can be obtained from the number of running instances of the task. In this way, the end time of the job and the amount of released resources can be estimated, and then the time when the resources meet the job requirements in the waiting queue can be estimated.
[0081] In some implementation manners, the value of the waiting time preset by the user may be parsed from the submission request of the first job; and the upper limit of the waiting time of the first job is determined according to the value of the waiting time. For example, the sum of the submission time and the waiting time may be used as the upper limit of the waiting time of the first job; for another example, the waiting time threshold may be preset to determine the waiting time threshold and the waiting time preset by the user The maximum value between the duration values, and the sum of the maximum value and the submission time of the first job as the upper limit of the waiting time of the first job. Of course, other methods can also be used to determine the upper limit of the waiting time of the first job in practical applications, and this article does not limit this.
[0082] In this embodiment, in order to ensure that the first job can at least be added to the waiting queue when submitted to the online mode, the waiting time threshold of the first job can also be preconfigured; the waiting time threshold of the first job can be determined according to the waiting time threshold. The lower limit of the waiting time; when the current available resources do not meet the requirements of the first job and the current time point is not greater than the lower limit of the waiting time of the first job, the first job is added to the waiting queue or kept The waiting state of the first job in the waiting queue.
[0083] In this embodiment, when the first time point is not greater than the upper limit of the waiting time of the first job, the waiting state of the first job in the waiting queue is maintained.
[0084] In some implementations, the first time point may be re-evaluated regularly; the first time point and the upper limit of the waiting time of the first job may be re-compared regularly, so as to adjust the waiting state of the job in the waiting queue in real time , Which is conducive to improving the processing efficiency and resource utilization of the distributed system.
