Dynamic priority iterator based on data characteristics in Gaia system

Abstract

The invention provides a dynamic priority iterator based on data characteristics in a Gaia system, and relates to the technical field of distributed big data calculation. The iterator comprises a priority scheduling module, a total iterative computation module and an incremental iterative computation module, the priority scheduling module reads the data of the data source to serve as an initial work set of iterative computation, and maintains a skip list for searching and selecting a data unit and a state list for storing state information corresponding to the data unit in each iterative taskexecution process; updating the state table according to a priority adjustment formula at the beginning of each iteration task, and determining the priority of each data unit after all the data unitsinput by the current iteration task are updated; the Gaia system performs iterative computation according to the priority information of each data unit; wherein the full iteration module is used for realizing iterative computation through a BulkIterate operator, and the increment iteration module is used for realizing iterative computation through a Delta Iterate operator.

Description

technical field [0001] The invention relates to the technical field of distributed big data computing, in particular to a dynamic priority iterator based on data characteristics in a Gaia system. Background technique [0002] The Gaia system is a new generation of big data computing system with high timeliness and scalability based on the hybrid coexistence of multiple computing models. Solve a series of key technical problems at several core levels of big data analysis systems such as adaptive and scalable big data storage, batch-flow fusion big data computing, high-dimensional large-scale machine learning, and high-time-effective big data intelligent interactive guides. Build an independent, controllable, time-effective and scalable new-generation big data analysis system, and master the core technology of the world's leading big data analysis system. [0003] In the big data environment, distributed iterative computing plays a vital role in data processing and analysis, ...

AI Technical Summary

Problems solved by technology

In large-scale data, only a small number of data update calculations will play a decisive role in the convergence of the iterative algorithm, and the role of most data update calculations is often limited, so the above iteration mechanism will cause Gaia to perform some different functions. The necessary calculations cannot make the iterative algorithm converge quickly, which greatly reduces the execution efficiency of the system

[0005] However, in today’s era of explosive data volume, the Gaia system’s processing efficiency for time-varying data is low, and the iterative algorithms supported begin to show many inapplicability, and system processing is also prone to bottlenecks, which cannot meet users’ needs for real-time performance.

Especially for data with data skew, if we still follow the traditional thinking and only consider the memory and performance of the processing nodes in the big data computing system to optimize the iterative computing process, without considering the characteristics of the iterative data being processed, it will be It will cause the system to perform some unnecessary operations and even prevent the iterative algorithm from converging within the effective time, which will affect the completion time and throughput of the overall task of the system

Images