Online graph division method for heterogeneous graph data

Abstract

The invention discloses a heterogeneous graph data-oriented online graph division method. The method comprises the following steps of: 1) evaluating the calculation speed imbalance and the storage space imbalance of a graph calculation system; determining the calculation speed imbalance of the graph calculation system according to the node function time complexity T of different types of nodes in the heterogeneous graph calculation of the graph calculation system; according to the storage space Sv occupied by the data carried by the different types of nodes and the storage space Se occupied by the data carried by the different types of edges in the heterogeneous graph calculation performed by the graph calculation system, determining the storage space imbalance of the graph calculation system; and 2) according to the node function time complexity T corresponding to different types of nodes, the storage space Sv and the storage space Se corresponding to different types of edges, distributing the current to-be-processed heterogeneous graph data to different partitions. According to the method, the task allocation in the graph calculation is optimized, so that the load and memory use in the graph calculation process are more balanced, and the result of improving the running efficiency of the graph calculation is achieved.

Description

technical field [0001] The invention belongs to the technical field of graph computing and real-time, and in particular relates to an online graph division method for heterogeneous graph data. Background technique [0002] Graph is an important data structure with more powerful expression ability, and can integrate different sources and different types of data into the same graph for analysis, and obtain results that are difficult to find by independent analysis. Therefore, graph computing can be widely used. It is widely used in social networks, recommender systems, network security, text retrieval, and biomedical fields. Many problems can be efficiently solved with the help of graph-related algorithms under the support of graph theory. In recent years, with the rapid development of technologies such as big data, machine learning, and data mining, the scale of abstracted graphs in many fields has grown exponentially. In order to cope with the challenges brought by massive d...

AI Technical Summary

Problems solved by technology

In the calculation process for heterogeneous graphs, the existing graph partitioning algorithm does not take into account the following problems in the processing of heterogeneous graphs: for such heterogeneous graphs, during the graph calculation process, the data stored by different types of nodes and edges The size of the occupied storage space may be different. For different node and edge types, different processing algorithms may be used, and the calculation time will also be different.

However, none of the existing graph partitioning algorithms are optimized for memory and computing load balancing for the characteristics of the above-mentioned heterogeneous graphs.

For example, some mainstream graph computing systems, such as GraphX, PowerGraph, etc. (refer to Gonzalez J E, Xin R S, Dave A, et al. Graphx: Graph processing in a distributed dataflow framework[C] / / 11th{USENIX}Symposium on Operating Systems Design and Implementation({OSDI}14).2014:599-613; Gonzalez JE,Low Y,Gu H, et al.Powergraph:Distributed graph-parallel computation on natural graphs[C] / / 10th{USENIX}Symposium on OperatingSystems Design and Implementation ({OSDI}12).2012:17-30), the graph partitioning algorithms used in it, such as hash partitioning, balanced partitioning, block partitioning, etc., do not take into account the calculation speed of heterogeneous graph nodes and edge data, The unbalanced characteristics of storage space lead to the low efficiency of the graph computing system in this case, and it is difficult to meet the real-time constraints brought to the computing process by the periodic arrival of massive data.

Images