A Storage-Optimized Distributed Graph Processing Method

Abstract

The invention discloses a storage optimization-based distributed graph processing method, and belongs to the field of graph calculation. The method comprises the following steps of: carrying out data partitioning in a data pre-processing stage; distributing graph partition data; starting data iteration processing; updating message transfer; making a work node extension decision; and ending the data processing. According to the method, a consistency hash algorithm is proposed to partition and store graph data, and an external storage mode-based distributed graph processing system is designed and realized; and by utilizing a dynamic storage optimization strategy, graph data processing load balance is realized according to partition storage of load adjustment graphs, so that the graph data processing speed is improved, the problem that the overall performance is reduced as hotspots are caused by load imbalance in the graph data processing process in the prior art is solved, and then the graph processing performance is improved.

Description

technical field [0001] The invention belongs to the field of graph computing, and more specifically relates to a storage-optimized distributed graph processing method. Background technique [0002] As a classic data structure, a graph expresses complex data relationships through points and edges. It has been widely used in various fields of society, including social data analysis and mining in the Internet field, protein interaction in the chemical field, and prediction of disease outbreak paths in the medical field. The citation relationship of literature in the academic field, etc., has derived many important algorithms, including PageRank, shortest path, connected branches, and extremely independent sets. Because graph data is of great significance and requires a lot of calculations, various graph processing systems have emerged. [0003] The first is the distributed memory mode graph processing system, including Pregel, GraphLab, etc. These systems first put all the inf...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the above defects of the prior art, the present invention provides a storage-optimized distributed graph processing method, which performs partition storage and IO balance on graph data, realizes graph data processing load balance, speeds up graph data processing, and solves problems existing in the prior art. Unbalanced load, causing hot spots in the process of graph data processing and causing overall performance degradation

Images