Graph iteration job-oriented running time prediction system and method in a Gia system

Abstract

The invention discloses a graph iteration job-oriented running time prediction system and method in a Gia system. The method comprises the following steps: quickly capturing off-line features of a current graph iterative algorithm through sampling execution before job execution, wherein the off-line features comprise convergence features and key input features of each iteration; continuously capturing runtime characteristics in a job execution process, wherein the runtime characteristics comprise job parameters, resource utilization conditions and detailed statistical data; taking the similarity between jobs as the basis of job matching and final predicted value calculation, mainly comprising static similarity captured through sampling execution and dynamic similarity captured through real execution. According to the matching algorithm, specific parameters of the algorithm can be trained through formulated similarity evaluation standards, so that iteration job can automatically adapt to various similarities. The invention is an end-to-end running time prediction method, integrates the offline features and the runtime features of the graph iteration job, and can accurately predict the running time of the distributed graph iteration job under low training overhead.

Description

technical field [0001] The invention relates to the technical field of distributed big data computing, in particular to a running time prediction system for graph-oriented iterative jobs in the Gaia system. Background technique [0002] The Gaia system is a new generation of big data computing system with high timeliness and scalability based on the hybrid coexistence of multiple computing models. Solve a series of key technical problems at several core levels of big data analysis systems such as adaptive and scalable big data storage, batch-flow fusion big data computing, high-dimensional large-scale machine learning, and high-time-effective big data intelligent interactive guides. Build an independent, controllable, time-effective and scalable new-generation big data analysis system, and master the core technology of the world's leading big data analysis system. [0003] Iterative computing is usually the core part of machine learning and graph processing algorithms, that...

AI Technical Summary

Problems solved by technology

Although these methods are quite effective, they are all limited to predicting under a fixed-configuration cluster, and because the obtained data are all offline data or rough statistical data, the prediction accuracy is not high

Images