Job execution method and device

Abstract

The invention provides a job execution method and device. The method comprises steps of collecting the information of a to-be-executed job, and the information comprises the priority information, the job state information, and the resource information of a client; inputting the information of the to-be-executed job into a pre-trained graph neural network model, and obtaining an execution score of the to-be-executed job and a client corresponding to the to-be-executed job; and then, according to the execution score of the to-be-executed job, executing the to-be-executed job on a client corresponding to the to-be-executed job. Thus, the execution scores of the to-be-executed jobs are acquired after the information of the to-be-executed jobs is inputted into the pre-trained graph neural network module, so the selection sequence of the to-be-executed jobs can be quickly decided in combination with dynamically-changed job influence factors, and selection accuracy of the to-be-executed jobs is improved; and then the to-be-executed jobs are sequentially executed on the clients corresponding to the to-be-executed jobs according to the selection sequence of the to-be-executed jobs, so resources are utilized to the maximum extent for job processing with high priority requirements, and big data processing requirements are met.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a job execution method and device. Background technique [0002] With the development of big data technology, the total amount of data has accumulated to the PB level. After the data processing process is split into job tasks, the number of job tasks has reached more than 100,000. The huge number of job tasks complicates job dependencies and increases Job selection is difficult, which can easily lead to long queuing time for jobs with high time-sensitive requirements, and the data cannot be completed within the specified time. At the same time, due to the limited resources of the scheduling client of the big data platform, it is impossible to satisfy all executable jobs to be executed at the same time, which will also increase the job queuing and execution time. [0003] Therefore, at present, jobs with higher timeliness requirements are given priority in execution, an...

AI Technical Summary

Problems solved by technology

[0002] With the development of big data technology, the total amount of data has accumulated to the PB level. After the data processing process is split into job tasks, the number of job tasks has reached more than 100,000. The huge number of job tasks complicates job dependencies and increases Job selection is difficult, which can easily lead to long queuing time for jobs with high time-sensitive requirements, and the data cannot be completed within the specified time

At the same time, due to the limited resources of the scheduling client of the big data platform, it is impossible to satisfy all executable jobs at the same time, which will also increase the job queuing and execution time.

[0003] Therefore, at present, jobs with higher timeliness requirements are given priority in execution, and jobs with higher timeliness requirements have higher priority to reduce the loss caused by job queuing and excessive execution time to a certain extent, but the priority of jobs is dynamic. In the process of change, jobs with lower priority may be raised to the highest level at special times

Therefore, relying only on job priority for job selection may result in jobs that process large amounts of data with a priority level that is not the highest level and cannot be completed under the timeliness requirements.

At the same time, with the continuous construction of the big data platform, the influencing factors affecting job selection may continue to change

Images