An edge partitioning method for graph partitioning

Abstract

The application provides an edge partition method for graph partition, which comprises the following steps: firstly, loading a graph to be partitioned and calculating a partition boundary value; then, unfolding the graph to obtain a conversion graph, creating and searching a set and a priority queue for the conversion graph; then, folding edges according to a greedy strategy, and partitioning subgraphs meeting the partition boundary value; finally, when the priority queue is empty, combining the remaining subgraphs to form a scale and partitioning. The application can achieve the advantages of simple partition and load balancing by actually using point partition and essentially using the edge partition method to partition the graph. In addition, the application folds edges under the greedy strategy, so that the edges adjacent in space tend to be folded in the same subgraph, thereby reducing the number of node replication times as much as possible and reducing communication overhead.

Description

Technical Field

[0001] This invention relates to the field of graph computation technology, and more specifically to an edge partitioning method for graph partitioning. Background Technology

[0002] In today's internet landscape, graphs are widely used in areas such as social networks, transportation networks, and computer vision. With the development of big data, artificial intelligence, and supercomputing, graph computing is increasingly demonstrating its value, while also facing greater challenges. Graph partitioning is a leading technique among many graph computing steps, serving as a prerequisite for achieving concurrent graph processing. The goals of graph partitioning are load balancing and reducing communication overhead; however, achieving both goals simultaneously is an NP-hard problem.

[0003] Currently, commonly used graph partitioning schemes can be divided into the following two types according to the partitioning objects:

[0004] 1. Graph partitioning: This method partitions the graph based on its nodes, distributing nodes evenly across multiple machine nodes by cutting edges. While relatively simple, this approach can easily lead to unbalanced partitioning, especially for power-law graphs.

[0005] 2. Graph edge partitioning: This method partitions the graph based on its edges, distributing the edges evenly across multiple machine nodes by splitting nodes. While more complex than node-based partitioning, it achieves better balance and requires less storage due to the absence of redundant edges.

[0006] The above solutions all have certain drawbacks: For the first method, on the one hand, it is easy to cause load imbalance, especially when a small number of nodes are connected to the majority of nodes. The machine nodes storing these few nodes will become the bottleneck for subsequent processing. On the other hand, storing redundant edges will increase space consumption. For the second method, the process of splitting nodes is relatively complicated. It is not possible to intuitively divide the edges connected to the node into multiple parts. Moreover, the node is copied during the node splitting process, which increases communication overhead. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a simpler edge partitioning method for graph partitioning that can minimize the number of node replications.

[0008] To solve the above-mentioned technical problems, the technical solution proposed by this invention is: an edge partitioning method for graph partitioning, comprising the following steps:

[0009] S1. Obtain the file of the graph to be decomposed, load the graph file, and calculate the boundary values ​​of the partition;

[0010] S2. Expand the graph obtained in step S1 with the edges as the center, and assign weights to each edge to obtain the transformation graph. At the same time, create a disjoint set and a priority queue for the transformation graph.

[0011] S3. Using a greedy strategy, select the edge with the smallest weight in the transformed graph obtained in step S2 and perform a folding operation. When the current subgraph satisfies the partition boundary value, partition and restore the subgraph.

[0012] S4. Repeat step S3 until the priority queue in step S2 is empty, that is, all edges have been dequeued and processed.

[0013] S5. When the priority queue in step S2 is empty, if there are subgraphs that do not meet the size requirements, they are combined according to the size relationship and the number of modules to be divided to form new subgraphs. The restored subgraphs are stored in a file to complete the division of all subgraphs.

[0014] Preferably, in the edge partitioning method for graph partitioning described above, step S1 includes the following steps:

[0015] S11. Obtain the file of the graph to be decomposed, encode each node, use an adjacency list to record the relationship between the preceding and following nodes of each edge, and count the number of edges in the graph.

[0016] S12. Divide the number of edges of the graph obtained in step S11 by the number of modules, and round down to obtain the boundary value of the partition.

[0017] Preferably, in the edge partitioning method for graph partitioning described above, step S2 includes the following steps:

[0018] S21. For the graph obtained in step S1, make a corresponding number of duplicate nodes according to the degree of each node, and assign an encoding to the duplicate nodes.

[0019] S22. Select unconnected duplicate nodes and connect them to all edges in the original graph, assigning these edges a weight of -1.

[0020] S23. Connect the duplicate nodes of the same node in sequence to form a loop, and assign the degree of the node as the weight to each edge of the loop to obtain the transformation graph;

[0021] S24. Create a disjoint set for each node of the transformation graph obtained in step S23. The result is stored in a two-dimensional vector. The first column of the two-dimensional vector stores the number of edges of the subgraph hidden by the current node, and the second column stores the nodes that contain the subgraph represented by the current node.

[0022] S25. Add the edges of the transformed graph obtained in step S23 to a priority queue according to their weights, so that the first element of the queue is the edge with the smallest weight.

[0023] Preferably, in the edge partitioning method for graph partitioning described above, step S3 includes the following steps:

[0024] S31. Use a greedy strategy to select the edge with the smallest current weight and perform a folding operation.

[0025] S32. After folding the edges selected in step S31, determine whether the current subgraph meets the partitioning requirements. If so, restore the partitioning of the subgraph and update the partitioning boundary values ​​according to the number of edges to be partitioned and the number of modules.

[0026] Preferably, in the edge partitioning method for graph partitioning described above, step S31 includes the following steps:

[0027] S311. Obtain the edge with the smallest current weight from the priority queue in step S25;

[0028] S312. For the two nodes of the edge obtained in step S311, query the disjoint set created in step S24 to obtain the outer nodes A and B that contain these two nodes respectively.

[0029] S313. Compare whether the two outer nodes A and B are the same node. If not, update the value of the second column of the row where the outer node A is located in step S24 to B, add A to the end of the row where the outer node B is located in the disjoint set, and update the value of the first column of the row where the outer node B is located in the disjoint set.

[0030] Preferably, in the edge partitioning method for graph partitioning described above, step S32 includes the following steps:

[0031] S321. Based on the boundary value obtained in step S12 or step S322, determine whether the number of edges of the subgraph contained in node A in step S313 satisfies the boundary value.

[0032] S322. If the number of edges in the subgraph contained in node B meets the boundary value, then the subgraph will be made invisible, that is, the value of the second column of its disjoint set will be set to -1, and the corresponding subgraph will be restored and stored in the corresponding file. The boundary value of the partition will be updated according to the number of edges to be partitioned and the number of modules.

[0033] Preferably, in the edge partitioning method for graph partitioning described above, step S5 includes the following steps:

[0034] S51. When the priority queue in step S2 is empty, if there are subgraphs that do not meet the size requirements, then arrange the subgraphs from smallest to largest according to the number of main edges they contain.

[0035] S52. Based on the subgraph sequence obtained in step S51 and the boundary values ​​obtained in step S12 or step S322, take out the largest subgraph, and then take out the smallest subgraph to be merged from the sequence and merge it with the current subgraph until the number of main edges of the current subgraph combination matches the boundary value.

[0036] S53. Based on the new subgraph obtained in step S52, restore it and store it in a file, and update the boundary values ​​of the partition according to the number of edges to be partitioned and the number of modules.

[0037] S54. Repeat steps S52 and S53 until the subgraph sequence obtained in step S51 is empty, thus completing the partitioning of all subgraphs.

[0038] Compared with existing technologies, the advantages of this invention are as follows: First, this invention essentially uses an edge partitioning method, thus enabling a more balanced partitioning of the graph into various partitioning modules, achieving load balancing. Furthermore, this invention expands the original graph in step S2 to obtain a transformed graph, and then folds the edges of the transformed graph into subgraphs according to the rules mentioned in this invention. Therefore, partitioning the subgraph is essentially partitioning the nodes, i.e., node partitioning. Thus, the partitioning difficulty is reduced after the transformation and folding. Simultaneously, this invention uses a greedy strategy to fold edges, so spatially adjacent edges tend to fold into the same subgraph, thereby minimizing the number of node replications and reducing communication overhead. Attached Figure Description

[0039] Figure 1 This is a flowchart of the partitioning method of the present invention.

[0040] Figure 2 This is the encoded image to be divided in a specific embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the unfolding process of the diagram to be divided in a specific embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of the folding process of the transformation diagram in a specific embodiment of the present invention.

[0043] Figure 5 This is a schematic diagram of the division result in a specific embodiment of the present invention. Implementation

[0044] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to the accompanying drawings and preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0045] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0046] refer to Figure 1 This invention provides an edge partitioning method for graph partitioning, the method comprising the following steps:

[0047] S1. Obtain the file of the graph to be decomposed, load the graph file, and calculate the boundary values ​​of the partition;

[0048] S2. Expand the graph obtained in step S1 with the edges as the center, and assign weights to each edge to obtain the transformation graph. At the same time, create a disjoint set and a priority queue for the transformation graph.

[0049] S3. Using a greedy strategy, select the edge with the smallest weight in the transformed graph obtained in step S2 and perform a folding operation. When the current subgraph satisfies the partition boundary value, partition and restore the subgraph.

[0050] S4. Repeat step S3 until the priority queue in step S2 is empty, that is, all edges have been dequeued and processed.

[0051] S5. When the priority queue in step S2 is empty, if there are subgraphs that do not meet the size requirements, they are combined according to the size relationship and the number of modules to be divided to form new subgraphs. The restored subgraphs are stored in a file to complete the division of all subgraphs.

[0052] Step S5 specifically includes the following steps:

[0053] S51. When the priority queue in step S2 is empty, if there are subgraphs that do not meet the size requirements, then arrange the subgraphs from smallest to largest according to the number of main edges they contain.

[0054] S52. Based on the subgraph sequence obtained in step S51 and the boundary values ​​obtained in step S12 or step S322, take out the largest subgraph, and then take out the smallest subgraph to be merged from the sequence and merge it with the current subgraph until the number of main edges of the current subgraph combination matches the boundary value.

[0055] S53. Based on the new subgraph obtained in step S52, restore it and store it in a file, and update the boundary values ​​of the partition according to the number of edges to be partitioned and the number of modules.

[0056] S54. Repeat steps S52 and S53 until the subgraph sequence obtained in step S51 is empty, thus completing the partitioning of all subgraphs.

[0057] First, the graph partitioning method of this invention essentially uses an edge partitioning method, thus enabling a more balanced distribution of the graph among various partitioning modules and achieving load balancing. Furthermore, this invention expands the original graph in step S2 to obtain a transformed graph, and then folds the edges of the transformed graph into subgraphs according to the rules mentioned in this invention. Therefore, partitioning the subgraph is essentially partitioning the nodes, i.e., node partitioning. Thus, the partitioning difficulty is reduced through transformation and folding. Simultaneously, this invention employs a greedy strategy for edge folding, so spatially adjacent edges tend to fold into the same subgraph, thereby minimizing the number of node replications and reducing communication overhead.

[0058] To more clearly illustrate the technical solution of the present invention, the overall detailed solution of the present invention will be described below in conjunction with a specific embodiment.

[0059] Specifically, step S1 in this invention includes the following steps:

[0060] S11. Obtain the file of the graph to be decomposed, encode each node, use an adjacency list to record the relationship between the preceding and following nodes of each edge, and count the number of edges in the graph.

[0061] S12. Divide the number of edges of the graph obtained in step S11 by the number of modules, and round down to obtain the boundary value of the partition.

[0062] Specifically, in this embodiment, such as Figure 2 As shown, the node codes A to F of the graph to be decomposed are shown. The number of edges in the graph is 7. In this embodiment, it is assumed that the number of partitioning modules is 2. After calculation, the partitioning boundary value is 3.

[0063] In this invention, step S2 specifically includes the following steps:

[0064] S21. For the graph obtained in step S1, make a corresponding number of duplicate nodes according to the degree of each node, and assign an encoding to the duplicate nodes.

[0065] S22. Select unconnected duplicate nodes and connect them to all edges in the original graph, assigning these edges a weight of -1.

[0066] S23. Connect the duplicate nodes of the same node in sequence to form a loop, and assign the degree of the node as the weight to each edge of the loop to obtain the transformation graph;

[0067] S24. Create a disjoint set for each node of the transformation graph obtained in step S23. The result is stored in a two-dimensional vector. The first column of the two-dimensional vector stores the number of edges of the subgraph hidden by the current node, and the second column stores the nodes that contain the subgraph represented by the current node.

[0068] S25. Add the edges of the transformed graph obtained in step S23 to a priority queue according to their weights, so that the first element of the queue is the edge with the smallest weight.

[0069] Specifically, in this embodiment, such as Figure 3 As shown, a corresponding number of duplicate nodes were created based on the degree of the original node AF, and these duplicate nodes were encoded using the same alphanumeric subscript format as the original node; then, as... Figure 3 As shown in the (ba) diagram, select the nodes that have not yet been connected from these copied nodes and connect them to form all the edges that appear in the original graph. These edges are called main edges, and a weight of -1 is assigned to the main edges; then, as... Figure 3 As shown in the (bb) diagram, the duplicate nodes generated by copying the same original node are connected in sequence to form a loop, and the degree of the original node is used as the weight to assign to each edge in the loop. These edges are called attached edges.

[0070] At this point, the unfolded transformation graph is obtained. This transformation graph can realize the transformation from edge partitioning to vertex partitioning, so that the partitioning result of the present invention has the advantage of load balancing of edge partitioning scheme, while the partitioning process can be as simple as vertex partitioning scheme.

[0071] After obtaining the transformed graph, a disjoint-set data structure is created for each node in the transformed graph. The result is stored in a two-dimensional vector. The first column of the two-dimensional vector stores the number of main edges of the subgraph contained in the current node, the second column stores the outermost node of the subgraph containing the current node, and the remaining columns represent the outermost nodes of the other subgraphs contained in the current node. For example: assuming the disjoint-set data structure is Q, then the i-th row of Q represents node i, Q[i][0] represents the number of main edges of the subgraph contained in node i; Q[i][1] = u means that the outermost node of the subgraph containing node i is u, and u = i if and only if node i is not contained in any other subgraph. If Q[i][1] = -1, it means that the subgraph has been partitioned; Q[i][j] = v (j = 2, 3, ...) means that node i contains the subgraph with the outermost node v.

[0072] The disjoint-set data structure obtained through step S24 of this invention stores both the path back from the current node to its top-level structure and provides the connection between the current node and its substructures. For example, for node i, its top-level structure is u = Q[i][1]. Then, based on Q[u][1], we obtain the top-level structure of u... until v = Q[v][1]. Node v is then the outermost node of the subgraph containing node i. By traversing the k-th (k>1) column of node i, we obtain the outermost nodes of all subgraphs contained by node i. Then, we traverse the m-th (m>1) column of node Q[i][k] to obtain its substructures... until all have been traversed.

[0073] Then, the edges of the transformed graph are added to a priority queue according to their weights, so that the first element of the queue is the edge with the smallest weight. Specifically, in this embodiment, as follows... Figure 3 As shown, there are edges with weights of -1, 2, 3 and 4 in the graph. Arranging these edges in ascending order will yield the advantage queue.

[0074] After obtaining the transformed graph and its corresponding disjoint-set data structure and priority queue, the edges are folded and partitioned. Step S3 in this invention specifically includes the following steps:

[0075] S31. Use a greedy strategy to select the edge with the smallest current weight and perform a folding operation.

[0076] S32. After folding the edges selected in step S31, determine whether the current subgraph meets the partitioning requirements. If so, restore the partitioning of the subgraph and update the partitioning boundary values ​​according to the number of edges to be partitioned and the number of modules.

[0077] Specifically, in this embodiment, it can be known that the edge with the smallest weight is the primary edge first, therefore the edge that is folded first is the primary edge, such as... Figure 4 The diagram (ca) shows a schematic of all the main edges after folding; after folding the main edges, the next step is to fold the secondary edges with a weight of 2, as shown in the diagram. Figure 4 The diagrams (cb), (cc), and (cd) shown in the figure are schematic diagrams after the corresponding attached edges have been folded.

[0078] In this invention, a greedy strategy is used to hide nodes with low degree as much as possible and guide nodes with high degree to be cut, thereby greatly reducing the number of node replications and reducing communication overhead.

[0079] After the above folding process is completed, it is determined whether the current subgraph meets the partitioning requirements. If the number of main edges contained in the current subgraph meets the partitioning boundary value, the subgraph is partitioned and restored, and the partitioning boundary value is updated according to the number of edges to be partitioned and the number of modules.

[0080] This invention enables partitioning at a fine-grained level by folding and discriminating simultaneously; at the same time, it updates the boundary values ​​after partitioning the subgraph, making the boundary values ​​more suitable for the remaining graphs to be partitioned, thereby improving the load balancing of the partitioning.

[0081] Specifically, step S31 in this invention includes the following steps:

[0082] S311. Obtain the edge with the smallest current weight from the priority queue in step S25;

[0083] S312. For the two nodes of the edge obtained in step S311, query the disjoint set created in step S24 to obtain the outer nodes A and B that contain these two nodes respectively.

[0084] S313. Compare whether the two outer nodes A and B are the same node. If not, update the value of the second column of the row where the outer node A is located in step S24 to B, add A to the end of the row where the outer node B is located in the disjoint set, and update the value of the first column of the row where the outer node B is located in the disjoint set.

[0085] Specifically, in this embodiment, Figure 4 Taking the (cc) graph as an example, the edge that needs to be folded is (E1, E2). Query the disjoint-set data structure created in step S24 to obtain the outermost nodes A and B containing these two nodes. That is, continuously backtrack the second column of these two nodes until Q[i][1] = i or Q[i][1] = -1. If Q[i][1] = -1, it means the current edge connects to a partitioned graph, so the edge does not need processing, and proceed to step S4. In this embodiment, the two nodes of edge (E1, E2) are E1 and E2, according to... Figure 4 As can be seen from the (ca) and (cb) diagrams, E1 is its outermost node, while E2 is contained within node F2.

[0086] Then, compare whether the two outer nodes obtained in step S312 are the same node. If not, update the value of the second column of the row containing outer node A in the disjoint-set obtained in step S24 to B, add A to the end of the row containing outer node B in the disjoint-set, and update the value of the first column of the row containing outer node B in the disjoint-set, which is the number of main edges of the subgraph contained in node B. Thus, the folding operation of one edge is completed. Figure 4 Taking the cc graph as an example, node E1 is hidden under node F2, i.e., Q[E1][1] = F2. E1 is merged into the back of F2, i.e., Q[F2].push_back( E1 ) . Finally, the number of main edges contained in F2 is updated, Q[F2][0] += Q[E1][0].

[0087] Specifically, step S32 in this invention includes the following steps:

[0088] S321. Based on the boundary value obtained in step S12 or step S322, determine whether the number of edges of the subgraph contained in node A in step S313 satisfies the boundary value.

[0089] S322. If the number of edges in the subgraph contained in node B meets the boundary value, then the subgraph will be made invisible, that is, the value of the second column of its disjoint set will be set to -1, and the corresponding subgraph will be restored and stored in the corresponding file. The boundary value of the partition will be updated according to the number of edges to be partitioned and the number of modules.

[0090] Specifically, in this embodiment, Figure 4Taking the (cd) graph as an example, after folding edge (F1, F2), the number of edges in the subgraph contained in F2 satisfies the partition boundary value of 3. Setting Q[F2][1] = -1 cancels the validity of edges associated with this subgraph, preventing them from participating in the folding operation. Then, iterate through the subgraph contained in F2, finding all main edges, i.e., edges with a weight of -1, which are (C4, F1), (E2, F2), and (C3, E1), and restoring them to (C, F), (E, F), and (C, E). After partitioning and restoring, we can obtain... Figure 5 The results are shown in the figure.

Claims

1. An edge partitioning method for graph partitioning, characterized in that, Includes the following steps: S1. Obtain the file of the graph to be decomposed, load the graph file, and calculate the boundary values ​​of the partition; Step S1 includes the following steps: S11. Obtain the file of the graph to be decomposed, encode each node, use an adjacency list to record the relationship between the preceding and following nodes of each edge, and count the number of edges in the graph. S12. Divide the number of edges of the graph obtained in step S11 by the number of modules, and round down to obtain the boundary value of the partition. S2. Expand the graph obtained in step S1 with the edges as the center, and assign weights to each edge to obtain the transformation graph. At the same time, create a disjoint set and a priority queue for the transformation graph. Step S2 includes the following steps: S21. For the graph obtained in step S1, make a corresponding number of duplicate nodes according to the degree of each node, and assign an encoding to the duplicate nodes. S22. Select unconnected duplicate nodes and connect them to all edges in the original graph, assigning these edges a weight of -1. S23. Connect the duplicate nodes of the same node in sequence to form a loop, and assign the degree of the node as the weight to each edge of the loop to obtain the transformation graph; S24. Create a disjoint set for each node of the transformation graph obtained in step S23. The result is stored in a two-dimensional vector. The first column of the two-dimensional vector stores the number of edges of the subgraph hidden by the current node, and the second column stores the nodes that contain the subgraph represented by the current node. S25. Add the edges of the transformed graph obtained in step S23 to a priority queue according to their weights, so that the first element of the queue is the edge with the smallest weight. S3. Using a greedy strategy, select the edge with the smallest weight in the transformed graph obtained in step S2 and perform a folding operation. When the current subgraph satisfies the partition boundary value, partition and restore the subgraph. Step S3 includes the following steps: S31. Use a greedy strategy to select the edge with the smallest current weight and perform a folding operation. S32. After folding the edges selected in step S31, determine whether the current subgraph meets the partitioning requirements. If so, restore the partitioning of the subgraph and update the partitioning boundary values ​​according to the number of edges to be partitioned and the number of modules. S4. Repeat step S3 until the priority queue in step S2 is empty, that is, all edges have been dequeued and processed. S5. When the priority queue in step S2 is empty, if there are subgraphs that do not meet the size requirements, they are combined according to the size relationship and the number of modules to be divided to form new subgraphs. The restored subgraphs are stored in a file to complete the division of all subgraphs.

2. The edge partitioning method for graph partitioning according to claim 1, characterized in that, Step S31 includes the following steps: S311. Obtain the edge with the smallest current weight from the priority queue in step S25; S312. For the two nodes of the edge obtained in step S311, query the disjoint set created in step S24 to obtain the outer nodes A and B that contain these two nodes respectively. S313. Compare whether the two outer nodes A and B are the same node. If not, update the value of the second column of the row where the outer node A is located in step S24 to B, add A to the end of the row where the outer node B is located in the disjoint set, and update the value of the first column of the row where the outer node B is located in the disjoint set.

3. The edge partitioning method for graph partitioning according to claim 1, characterized in that, Step S32 includes the following steps: S321. Based on the boundary value obtained in step S12 or step S322, determine whether the number of edges of the subgraph contained in node A in step S313 satisfies the boundary value. S322. If the number of edges in the subgraph contained in node B meets the boundary value, then the subgraph will be made invisible, that is, the value of the second column of its disjoint set will be set to -1, and the corresponding subgraph will be restored and stored in the corresponding file. The boundary value of the partition will be updated according to the number of edges to be partitioned and the number of modules.

4. The edge partitioning method for graph partitioning according to claim 3, characterized in that, Step S5 includes the following steps: S51. When the priority queue in step S2 is empty, if there are subgraphs that do not meet the size requirements, then arrange the subgraphs from smallest to largest according to the number of main edges they contain. S52. Based on the subgraph sequence obtained in step S51 and the boundary values ​​obtained in step S12 or step S322, take out the largest subgraph, and then take out the smallest subgraph to be merged from the sequence and merge it with the current subgraph until the number of main edges of the current subgraph combination matches the boundary value. S53. Based on the new subgraph obtained in step S52, restore it and store it in a file, and update the boundary values ​​of the partition according to the number of edges to be partitioned and the number of modules. S54. Repeat steps S52 and S53 until the subgraph sequence obtained in step S51 is empty, thus completing the partitioning of all subgraphs.

Images