An online graph partitioning method for heterogeneous graph data

Abstract

This invention discloses an online graph partitioning method for heterogeneous graph data, the steps of which include: 1) evaluating the computational speed imbalance and storage space imbalance of the graph computing system; determining the computational speed imbalance of the graph computing system based on the node function time complexity T of different types of nodes in the heterogeneous graph computation; and determining the storage space S occupied by the data carried by different types of nodes in the heterogeneous graph computation based on the graph computing system. v The storage space S occupied by the data carried by different types of edges e 1) Determine the memory imbalance of the graph computation system; 2) Based on the time complexity T and memory S of the node functions corresponding to different types of nodes. v The storage space S corresponding to different types of edges e This invention distributes the currently pending heterogeneous graph data across different partitions. It optimizes task allocation in graph computation, resulting in a more balanced load and memory usage during computation, thereby improving the efficiency of graph computation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of graph computing and real-time, and particularly relates to an online graph partitioning method for heterogeneous graph data. BACKGROUND

[0002] A graph is an important data structure, has stronger expression ability, and can fuse different sources and different types of data into the same graph for analysis, so as to obtain results that cannot be found by independent analysis. Therefore, graph computing can be widely applied in social networks, recommendation systems, network security, text retrieval, biological medicine and other fields, and many problems can be efficiently solved by means of graph-related algorithms under the support of graph theory. In recent years, with the rapid development of big data, machine learning and data mining technologies, the scale of graphs abstracted in many fields grows exponentially. In order to cope with the challenges brought by massive data, distributed graph computing systems are now widely used. Graph partitioning algorithm is an important part of distributed graph computing system, which divides a graph into several subgraphs for running in a distributed system, and distributes the point and edge data on the subgraph and the computing task on the subgraph to each node. After the distribution is completed, the iterative computing task of graph computing is run in parallel in the distributed system until there is no new node or edge, and the computing task of all nodes is completed, so as to complete the whole graph computing process. As can be seen, an efficient graph partitioning algorithm can significantly improve the running performance of the graph computing system.

[0003] In many applications, massive data often arrives in batches periodically, for example, continuous power flow calculation in power grid. At this time, the graph computing process is triggered periodically and has time constraints. The batch-arriving data must be processed within the current period, otherwise, the data will be covered by the data arrived in the next period, resulting in calculation errors due to data loss. Therefore, graph partitioning must also be performed periodically online, and the graph computing process should be completed within the time constraints as much as possible.

[0004] A heterogeneous graph is a kind of graph widely existing in the real world, which is a graph with multiple node types or edge types. For example, a paper citation network is a typical heterogeneous graph. It has author nodes and paper nodes, and the edges have multiple types, such as co-authorship edges between author nodes, affiliation edges between author nodes and paper nodes, citation edges between paper nodes, and so on. In the process of computing heterogeneous graphs, the existing graph partitioning algorithms do not consider the following problems in the processing of heterogeneous graphs: for such heterogeneous graphs, the data stored in different types of nodes and edges may occupy different sizes of storage space, and different processing algorithms may be used for different types of nodes and edges, and the computing time will also be different. However, the existing graph partitioning algorithms do not optimize the memory and computing load balancing for the above characteristics of heterogeneous graphs. For example, some mainstream graph computing systems, such as GraphX, PowerGraph, etc. (see Gonzalez J E, Xin R S, Dave A, et al. Graphx: Graph processing in a distributed dataflow framework [C] / / 11th {USENIX} Symposium on Operating Systems Design and Implementation (OSDI 14). 2014: 599-613; Gonzalez JE, Low Y, Gu H, et al. Powergraph: Distributed graph-parallel computation on natural graphs [C] / / 10th {USENIX} Symposium on Operating Systems Design and Implementation (OSDI 12). 2012: 17-30), the graph partitioning algorithms used by them, such as hash partitioning, balanced partitioning, block partitioning, etc., do not consider the characteristics of the computing speed and storage space imbalance of heterogeneous graph nodes and edges, resulting in low efficiency of the graph computing system in this case, and further difficult to meet the real-time constraints brought by the periodic arrival of massive data to the computing process. SUMMARY

[0005] In view of the shortcomings of the existing graph partitioning technology, the purpose of the present application is to provide an online graph partitioning method for heterogeneous graph data, namely OGP-HG (Online Graph Partitioning for Heterogeneous Graph), which can complete the graph partitioning for online heterogeneous graph data in a distributed environment, thereby optimizing the existing task allocation process in graph computing, making the calculation load and memory usage of the graph computing process more balanced, and achieving the result of improving the running efficiency of graph computing.

[0006] The online graph partitioning method for heterogeneous graph data of the present application comprises the following steps:

[0007] 1. Evaluate the calculation speed imbalance and storage space imbalance of the heterogeneous graph. The graph computing system in the present application adopts the GAS model, namely the Gather-Apply-Scatter model. The calculation speed imbalance of the heterogeneous graph partitioning mainly exists in the Apply phase, namely the calculation phase after the node gathers the messages, and its calculation load is the node function running on the node, so evaluating the calculation speed imbalance of the heterogeneous graph is to obtain the time complexity (denoted as T) of the node function of different types of nodes in the heterogeneous graph calculation.

[0008] The OGP-HG graph partitioning algorithm in the present application belongs to the point partitioning algorithm, each edge is only allocated to one partition, and a node may have a copy in multiple partitions. Therefore, evaluating the storage space imbalance of the heterogeneous graph is to obtain the storage space (denoted as S v and S e ) occupied by the data carried by different types of points and edges in the heterogeneous graph calculation.

[0009] In order to perform graph partitioning according to the calculation speed imbalance and storage space imbalance of the heterogeneous graph, the node function and data amount of the node need to be obtained before graph partitioning. The types of nodes are limited, each type of node has its corresponding data structure and node function, so the values of T and S v are limited. T, S v and S e can be directly given in the form of parameters by the user, and if not given, they can be evaluated by the graph computing system in the present application before graph partitioning. The time complexity of the node function needs to be obtained in advance, and the average value of 100 times of its use in the system is taken as the time complexity (T). The obtained T and S v are stored in the form of key-value pairs of (VertexType, T) and (VertexType, S v ). The types of edges can be unlimited, and the storage space S e occupied by the data carried by the edges can be obtained in real time before graph partitioning.

[0010] 2. Use T, S of different types of nodes obtained in step 1 v S of different types of edges e Assign the input online heterogeneous graph data to different partitions, first define the graph G = (V, E), V = (v1,...v n ) is the point set, E = (e1,...e m ) is the edge set, the specific steps are as follows:

[0011] 2.1 Heterogeneous graph data comes in the form of point data and edge data in turn, whenever a piece of edge data is divided, its form is like (SrcId, DstId, EdgeAttr1, EdgeAttr2,...), where SrcId and DstId are the two node Ids of the edge, EdgeAttr1 is the first attribute data of the edge, this edge can be written as e = (v src ,v dst ), that is, the edge e is represented by its starting node v src and the ending node v dst . Define P = (p1, p2,...p i ) to represent the set of all partitions, A(v) to represent the set of partitions that already have a copy of node v, and δ(v) to represent the partial degree of node v on the current partition (i.e. how many adjacent edges node v has on the current partition). A greedy algorithm based on the improved HDRF (reference Petroni F, Querzoni L, Daudjee K, et al. Hdrf: Stream-based partitioning for power-law graphs [C] / / Proceedings of the 24th ACM international on conference on information and knowledge management. 2015: 243-252.) is used to divide the edge data, that is, for a piece of edge data, calculate its score when assigned to each partition p ∈ P Where the replication score Balancing score Where the parameter μ is used to control the degree of imbalance on different partitions, when μ = 0, the balance of the partition is completely ignored, the larger the μ, the higher the degree of influence of the balance of the partition on the algorithm. maxsize and minsize in OGP-HG algorithm refer to the maximum and minimum storage space occupied by all partitions, |p| is the storage space occupied by the data already in p partition. Traverse all partitions, find the one that makes the score C SGP-HG(v src ,v dst The partition p that yields the maximum value is denoted as p_i. max That is, the edge will be assigned to p max On the partition, after each edge is assigned, update |p| and A(v) based on the assignment result. src ), A(v dst ), δ(v src ), δ(v dst ).

[0012] 2.2 Whenever a received point data is processed, its format is (VertexId, VertexType, VertexAtrr1, VertexAtrr2, ...), where VertexId is the node ID, VertexType is the node type, and VertexAtrr is the node attribute data. The time complexity T of the node function obtained based on VertexType is used. A greedy algorithm is employed to allocate nodes, ensuring balanced computational load across different partitions after node allocation. Therefore, the following is defined: Where maxtime and mintime represent the maximum and minimum function execution times of the allocated nodes within all partitions, respectively, p t This indicates the time consumed by the node functions of the allocated nodes within partition p; this invention iterates through all partitions p... t Find out p that gets the maximum value t This involves assigning node v to partition p, which has the lowest current computational load. By assigning each arriving node sequentially, the computational load of the node function is evenly distributed across the partitions. After each node is assigned, the corresponding partition p is updated based on the assignment result. t .

[0013] 2.3 After each batch of data is allocated, the memory usage of all partitions is calculated as |p1|, |p2|, ..., |p i The maximum and minimum values ​​of | are used to update maxsize and minsize. Similarly, the time p of the function is calculated based on the total number of partition nodes. t1 p t2 ...p ti Update maxtime and mintime with the maximum and minimum values.

[0014] 2.4 The divided nodes and edges form a graph, and the graph computation iteration operation is started. In the graph computation, the node collects information of all adjacent nodes and adjacent edges, and performs computation according to its node function to obtain a computation result. When the difference between the computation results of two iterations is less than a set convergence value, it is considered that the node has reached convergence, and the node will not be computed in the next iteration. When new nodes and edges arrive, the graph structure is updated, and part of the previously converged nodes are activated according to the activation function, so that these nodes rejoin the computation. The specific part of the nodes to be activated is determined by the user-defined activation function.

[0015] 2.5 Repeat steps 2.1-2.4 until no new nodes or edges arrive, and all nodes reach convergence, at which time the graph division ends.

[0016] 3. In step 2.1, different data structures are used to maintain |p|, A(v), δ(v) and the like according to the edge allocation result, and the specific manner is as follows:

[0017] 3.1 |p| is maintained on each partition p. When an edge e=(v src ,v dst ) is allocated on partition p, one edge e and its two nodes v src , v dst will be added to partition p, so that |p| increases the data amount of the edge and its two nodes. When a node v has a copy on partition p, the newly arrived edge will not generate a copy of v on partition p, so |p| = mem(e) + α*mem(v src ) + β*mem(v dst ), where A(v) represents a set of partitions in which a copy of node v already exists, mem(e) represents the memory space occupied by edge e, and mem(v) represents the memory space occupied by node v.

[0018] 3.2 A(v) is maintained using a set. After allocating an edge e=(v src ,v dst ) to partition p, A(v src ) = A(v src ) ∪ {p}, and A(v dst ) = A(v dst ).

[0019] 3.3 A HashMap is used to maintain δ(v) on each partition. After allocating an edge e=(v src ,v dst ) to partition p, δ(v src ) = δ(v src ) + 1 on partition p, and δ(v dst ) = δ(vdst )+1.

[0020] Compared with the prior art, the positive effects of the present application are:

[0021] The patent proposes a new online graph partitioning algorithm based on heterogeneous graphs. The online graph partitioning problem is a new derivative of the graph partitioning problem. Compared with traditional graph partitioning algorithms, the online graph partitioning algorithm does not store the complete graph on the computer executing the partitioning algorithm, but performs online partitioning on the continuously read-in edges or points, processes only part of the graph data at a time, and allocates the part of the graph data to the nodes, and then no longer migrates the already allocated data. Since it is not necessary to store the complete graph data on a single computer, the online graph partitioning algorithm can be applied to larger scale graphs compared with traditional graph partitioning algorithms, and has the advantages of low time complexity and the ability to process dynamic data. At the same time, the online graph partitioning algorithm in the patent is based on heterogeneous graphs, and can realize load balancing and memory optimization for heterogeneous graphs. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the structure of a graph to be partitioned in the present application.

[0023] Figure 2 is the partitioning of subgraphs and nodes of the graph on different partitions by edge and node copies.

[0024] Figure 3 is the graph partitioning and graph computing flowchart in the present application. DETAILED DESCRIPTION

[0025] The technical solutions of the present application will be described in detail below in combination with embodiments and drawings. It can be understood that the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0026] Figure 3 is the graph partitioning and graph computing flowchart in the present application, and the present application will be described below in combination with a heterogeneous graph containing three types of node functions and edges carrying different data amounts. The point data of the already arrived heterogeneous graph:

[0027] vertexId vertexType vertexAttr1 vertexAttr2 vertexAttr... 1 3 0 0 ... 2 2 0 0 ... 3 2 0 0 ... 4 1 0 0 ... 5 1 90 30 ... 6 1 0 0 ... 7 1 100 35 ... 8 1 0 0 ...

[0028] It is assumed that the time complexity of the node function (vertexType, T) has been evaluated as (1, 1), (2, 3), and (3, 5). The node data occupies space (vertexType, S v ) is (1, 16), (2, 24), and (3, 32).

[0029] The edge data of the heterogeneous graph that has already arrived and its data space S:

[0030]

[0031]

[0032] First, partition the edge data, let In this context, μ and ε are both set to 3.1. Dividing these 11 edges into 3 partitions P = {p1, p2, p3} results in:

[0033] p1 p2 p3 (1,4) (4,5) (5,6) (7,8) (3,6) (6,7) (8,4) (8,2) (8,5) (8,3) (8,6)

[0034] The numerical calculations during the partitioning process are shown below:

[0035] (1) The process of data volume change in the three partitions as the edges are divided

[0036] srcId dstId |p1| |p2| |p3| 1 4 0 0 0 4 5 84 0 0 5 6 84 72 0 3 6 84 72 96 6 7 84 144 96 7 8 84 144 172 8 2 172 144 172 8 3 172 220 172 8 4 172 284 172 8 5 300 284 172 8 6 300 284 232

[0037] (2) The process of how the set of partitions containing the replicas of all nodes changes as the edges are partitioned.

[0038]

[0039]

[0040] (3) The process of how the partial degree of all nodes changes with the partitioning of edges.

[0041] srcId dstId δ(v1) δ(v2) δ(v3) δ(v4) δ(v5) δ(v6) δ(v7) δ(v8) 1 4 0 0 0 0 0 0 0 0 4 5 1 0 0 1 0 0 0 0 5 6 1 0 0 2 1 0 0 0 3 6 1 0 0 2 2 1 0 0 6 7 1 0 1 2 2 2 0 0 7 8 1 0 1 2 2 3 1 0 8 2 1 0 1 2 2 3 2 1 8 3 1 1 1 2 2 3 2 2 8 4 1 1 2 2 2 3 2 3 8 5 1 1 2 3 2 3 2 4 8 6 1 1 2 3 3 3 2 5

[0042] (4) From the data volume in (1), we can obtain the allocation of each edge e = (v src ,v dst Balanced score across 3 partitions at time )

[0043]

[0044]

[0045] (5) From the partition set where the edge replicas are located and the partial degree of the nodes in (2) and (3), we can obtain the value of e = (v) for each edge. src ,v dst Replication score across 3 partitions

[0046] srcId dstId C REP 1]]> C REP 2]]> C REP 3]]> 1 4 0 0 0 4 5 1 0 0 5 6 0 1 0 3 6 0 0 1 6 7 0 1 1 7 8 0 0 1 8 2 1 0 0 8 3 0 3 0 8 4 3 3 0 8 5 1.333333333 3 1.666666667 8 6 1.375 3 3

[0047] (6) The total score C of the OGP-HG partitioning algorithm in this invention is obtained by summing the replication score and balance score in (4) and (5).SGP-HG C SGP-HG The partition that reaches the maximum is the partition to which the edge is to be assigned.

[0048]

[0049]

[0050] The allocation calculation process of the 11th edge is described in detail below: when the 11th edge is allocated, the balance score is calculated Where μ = ε = 3.1, maxsize = 300, minsize = 232, |p1| = 300, |p1| = 300, |p1| = 300, so we get:

[0051] C BAL 1]]> C BAL 2]]> C BAL 3]]> 0 0.697609001 2.964838256

[0052] The replication score is calculated Where A(v6) = {p2, p3}, A(v8) = {p1, p2, p3}, δ(v6) = 3, δ(v8) = 5, so we get:

[0053] C REP 1]]> C REP 2]]> C REP 3]]> 1.375 3 3

[0054] From We get:

[0055] C1 C2 C3 1.375 3.697609001 5.964838256

[0056] C3 is the largest, so the edge (8, 6) should be assigned to partition p3.

[0057] Next, the node data is divided, according to the scoring function The incoming node is assigned to the partition with the minimum node function calculation time, so the allocation result is:

[0058] partition p1 p2 p3 allocation node 1,8 2,4,6 3,5,7 node function time 6 5 5

[0059] At this time, it can be observed that the node function calculation time of the three partitions is close, achieving the division effect. Figure 2 The incoming graph data is divided into subgraphs and nodes according to the OGP-HG algorithm.

[0060] After the graph data is divided, the graph is composed according to the division result and the graph calculation is performed. When new graph data arrives, the graph division is continued until there is no new node or edge, and all nodes reach convergence. At this time, it is considered that the graph calculation is completed.

[0061] The above examples are merely illustrative of the principles of the application and its best mode and are not intended to limit the scope of the application. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with any other feature. Any feature described herein can be implemented in any combination with

Claims

1. An online graph partitioning method for heterogeneous graph data, comprising the steps of: 1) evaluating a computation speed imbalance and a storage space imbalance of the graph computing system; wherein the computation speed imbalance of the graph computing system is determined according to a node function time complexity T of different types of nodes in the heterogeneous graph computation by the graph computing system; the storage space imbalance of the graph computing system is determined according to a storage space S occupied by data carried by different types of nodes in the heterogeneous graph computation by the graph computing system v and a storage space S occupied by data carried by different types of edges e ​ 2) the node function time complexity T and storage space S corresponding to different types of nodes v the storage space S corresponding to different types of edges e allocating the current to-be-processed heterogeneous graph data to different partitions according to the method of steps 21) to 24); 21) Whenever a cut is made in a heterogeneous graph G, an edge e = (v src ,v dst ) is first assigned a score where the replication score Balanced score ε is taken as 3.1, A(v) represents the partition set of which node v has already been in, δ(v) represents the partial degree of node v on the current partition, v∈(v src ,v dst ), v src is the starting node of edge e, v dst is the terminal node of edge e, parameter μ is used to control the degree of imbalance on different partitions; maxsize and minsize refer to the maximum and minimum storage space occupied in the current all partitions, |p| is the storage space occupied by the existing data on partition p; traverse all partitions in the partition set P, find the partition p that makes the maximum value of C SGP-HG (v src ,v dst ,p) and mark it as p max , then distribute the current edge e to the p max partition, and then update |p|, A(v src ), A(v dst ), δ(v src ), δ(v dst ) according to the distribution result; 22) Whenever a node v in the heterogeneous graph G is partitioned, the node function time consumption of all the partitions in the assignment set P is traversed, and the partition to which the current node v is assigned is determined to be the one that makes the maximum; then the node function time consumption of the corresponding partition is updated according to the assignment result; wherein, maxtime and mintime represent the maximum and minimum node function time spent by allocated nodes within all partitions in the allocation set P, p t node function time spent by allocated nodes within partition p; 23) Assembling the partitioned nodes and edges into a graph, and starting the graph computation iteration; 24) Repeating steps 21) to 23) until all edges and nodes of the heterogeneous graph G are processed, and the partitioning of the heterogeneous graph G is completed.

2. The method of claim 1, wherein, |p| = mem(e) + a * mem(v src ) + b * mem(v dst ), where A(v) denotes the set of partitions that already have a copy of node v, mem(e) denotes the memory space occupied by edge e, and mem(v) denotes the memory space occupied by node v.

3. The method of claim 1, wherein, A(v) is maintained using a set, when an edge e = (v src ,v dst ) is assigned to a partition p, A(v src ) = A(v src ) U {p}, A(v dst ) = A(v dst ) U {p}.

4. The method of claim 1, wherein, Maintain δ(v) using HashMap on each partition, when edge e = (v src ,v dst ) is assigned to partition p, δ(v src ) on p partition = δ(v src ) + 1, δ(v dst ) = δ(v dst ) + 1.

5. The method of claim 1, wherein, The graph computation system is a GAS model, which comprises three stages of Gather, Apply, and Scatter in sequence.

6. A server, characterized by A computer program product comprising a memory storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the steps of any of the methods of claims 1 to 5.

7. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program product, when executed by the processor, implements the steps of any of the methods of claims 1 to 5. The computer program product, when executed by the processor, implements the steps of any of the methods of claims 1 to 5.

Images