Large-scale graphical partition method based on vertex cut and community detection

Abstract

The invention discloses a multilayer k-way graphical partition method based on vertex cut and community detection. The method comprises the steps that the distribution of a natural graph is considered according to the statistic analysis property, a corresponding vertex cutting algorithm is provided, vertexes causing longer task completion time are cut, label propagation is iteratively performed on the cut graph by a community detection algorithm based on the label propagation, the label of each vertex of the graph is determined, a community where the vertexes are located is obtained, partitioning is performed by a traditional multilayer k-way graph partition algorithm, and the efficiency is consolidated. For most of application in the large-scale iteration graph processing, distributed computational nodes meet the load balancing, extra communication traffic, due to the iteration dependency necessity, produced by each processing original node between adjacent iteration processing steps is greatly reduced, the task operating efficiency of a graph processing frame is greatly reduced, and the throughput capacity of tasks is increased.

Description

technical field [0001] The invention belongs to the technical field of computer science, and more specifically relates to a large-scale graph partitioning method based on vertex cutting and community gathering. Background technique [0002] With the development of computer technology and the wide application of Web2.0, the amount of data in the Internet is becoming larger and larger, and there are more and more challenges to the processing of these data, one of which is the processing of massive graph data (Fig. calculation), such as PageRank calculation for massive webpage data, social relationship analysis in social networks, network document relationship analysis, etc., because the main feature of graph calculation is that it needs multiple iterations, and the calculation units need to communicate with each other, so the traditional The full-scale computing framework MapReduce is not suitable for graph computing, so a number of dedicated large-scale graph computing framew...

AI Technical Summary

Problems solved by technology

[0003] In a distributed environment, the graph partition algorithm in the graph computing processing framework directly affects the processing efficiency of the framework. The existing frameworks all use a simple Hash algorithm. Although it is simple and fast, it can only satisfy load balancing. The traditional MGP (Multilevel Graph Partition) partition scheme has insufficient processing performance for natural graphs, because it cannot take into account the Power-Law distribution characteristics of natural graphs, so the partition efficiency is not high, and bottlenecks often occur in the communication traffic of nodes during iterations, which greatly affects tasks. Completion time, which in turn affects the computing performance and service quality of the overall platform

[0004] Of course, with the deepening of research, some emerging solutions have emerged, such as streaming-based solutions, which abstract the loading of graphs into incremental streaming data, and use some simple heuristic partitioning algorithms to fully consider the graph partitioning. The minimum edge cut and vertex equalization implement different algorithms, but cannot solve the Power-Law

Images