A Graph Segmentation Method for Distributed Graph Computing

Abstract

Aiming at the technical problem that the existing graph segmentation method is too expensive for large-scale graph calculation, the present invention provides a graph segmentation method for distributed graph computing, based on the open source framework PowerLyra, first as the input The graph data distributes the edges to each node according to the hash of the target point, and then each node processes the edges distributed to its nodes synchronously and in parallel on the distributed framework, and executes the corresponding distributed algorithm as needed. The present invention proposes the concept of "package", which refers to a collection of partial edges with high locality, and its maximum value can be modified. Through the package, the data locality characteristic is introduced into the measurement standard of graph division, so that the divided subgraph It has locality, making full use of the principle of data locality, effectively reducing the number of mirror vertices of vertices, and at the same time ensuring load balancing between nodes, reducing communication overhead in distributed systems, and improving the processing efficiency of large-scale graph-related applications. Improve the performance of distributed graph computing.

Description

technical field [0001] The invention belongs to the field of distributed graph computing, in particular to a graph segmentation method for distributed graph computing. Background technique [0002] There are currently many distributed graph computing frameworks developed around the world, mainly including two parts: graph partitioning and graph computing. Graph partitioning refers to distributing large-scale graph data to each node in the cluster according to a certain strategy, and the graph computing part refers to performing distributed operations on the graph according to the needs of practical applications. [0003] In the early days, there was PowerGraph from Carnegie University, which introduced graph division from edge segmentation to the era of point segmentation. Before this, the principle of graph partitioning is that each point exists on only one node, and all edges related to this point are stored on this node, and each edge is stored twice. In practical appli...

AI Technical Summary

Problems solved by technology

[0006] Aiming at the technical problem that the existing graph segmentation method is too expensive for large-scale graph calculation, the present invention provides a distributed The graph segmentation method of graph computing, referred to as the TopoX partition method. The TopoX partition method is based on the open source framework PowerLyra. First, the input graph data is distributed to each node, and input from each node to the entire distributed framework for processing.

Images