Computing cluster job scheduling method and device, computer equipment and storage medium

Abstract

This invention belongs to the field of job scheduling, and relates to a method, apparatus, computer equipment, and storage medium for scheduling jobs in a computing cluster. The method includes: acquiring multi-source job data; preprocessing and constructing multi-dimensional features from the multi-source job data; constructing a heterogeneous ensemble learning model and performing quantile regression training; based on the quantile regression training results, quantifying prediction uncertainty and generating job runtime estimates and probabilistic prediction intervals; transforming the job runtime estimates and probabilistic prediction intervals into a set of budget uncertainties and establishing a robust optimization scheduling model; transforming the robust constraints in the model into equivalent deterministic linear constraints; embedding the robust optimization scheduling model into a rolling time-domain control framework, and combining it with a prediction error feedback mechanism to achieve online adaptive scheduling. This achieves the proactive quantification and utilization of job runtime prediction uncertainty; improves the reliability and performance stability of the system under uncertain environments; and enhances reliability.

Images