Acceleration method for maximum flow of large-scale graph parallel computing based on cut-point partitioning mechanism

Abstract

The present invention relates to an acceleration method for parallel calculation of maximum flow of large-scale graphs based on a cut point partitioning mechanism. The key steps include: 1) constructing a coverage map of the large-scale graph; 2) determining the unique path from the source point to the sink point on the coverage map; 3) submitting all subgraphs corresponding to the vertices on the path to the GraphChi platform for parallel maximum flow calculation and integration of the subgraph maximum flow. The present invention ensures the independence of the maximum flow calculation of each subgraph, reduces the number of communications, and integrates the results by parallel calculation of the maximum flow to ensure its computational efficiency and correctness. It can solve the maximum flow problem of large-scale graphs and significantly improve the calculation speed of solving the maximum flow. Experimental results show that this method can make full use of the characteristics of the problem itself to cope with the scenario of calculating the maximum flow of large-scale graphs. Compared with classical algorithms, it can effectively accelerate the calculation speed of the maximum flow.

Description

Technical Field

[0001] The present invention relates to an acceleration method for large-scale graph maximum flow calculation, which is a method for realizing fast maximum flow calculation on a parallel graph computing framework GraphChi in combination with a cut point segmentation algorithm. Background Art

[0002] With the continuous development of science and technology, the data generated in the fields of transportation, information services, telecommunications, etc. are growing at an explosive rate. Usually, these data are presented in the form of large-scale graphs. Many practical problems such as the maximum traffic volume of urban traffic, the transaction information transmission carrying capacity of users on Double Eleven, and the load of the number of simultaneous calls in a region can be transformed into maximum flow problems. The maximum flow problem in large-scale graphs has become an important research direction in the graph theory system. After years of efforts, the existing maximum flow problem in the network has gradually become more perfect, but the efficiency of solving the maximum flow of large-scale graphs is low and still cannot meet the needs of many current application scenarios.

[0003] In the current field of big data analysis, the amount of graph data that needs to be processed is often as high as billions. The graph data structure is complex and changeable, and the coupling is strong. Therefore, this type of calculation usually requires a large amount of memory and appears on a high-performance computing cluster, which greatly increases the development cost, and the calculation speed still cannot reach the optimal. Similarly, the distributed graph computing system requires timely message communication between each program. In large-scale graph computing, a large amount of data requires more frequent communication. Such time-consuming operations cannot efficiently perform parallel computing of multiple subgraphs. The high network infrastructure cost and information loss also limit its application. The disk-based stand-alone graph computing system GraphChi solves this problem well. GraphChi uses the hard disk on the personal computer to store the graph on the hard disk. In addition, GraphChi uses parallel sliding window to reduce random read and write of the disk, thereby greatly shortening the disk access time during the processing process. The GraphChi platform can be used to calculate the maximum flow of the subgraph in parallel, which greatly improves the speed of the entire calculation process. Summary of the invention

[0004] (I) Technical solutions

[0005] The main purpose of this invention is to solve the problem of low efficiency of existing methods for solving the maximum flow of large-scale graphs, and propose a maximum flow acceleration method for large-scale graph calculations based on the cut point segmentation mechanism on the parallel framework GraphChi to improve the calculation efficiency. This method mainly utilizes the parallelism of the GraphChi framework and the efficient use of the hard disk by the GraphChi framework to significantly improve the calculation speed of solving the maximum flow of large-scale graphs and reduce the calculation complexity.

[0006] (II) Technical solution

[0007] In order to explain the present invention more clearly, the relevant terms involved in this article are explained as follows:

[0008] Large-scale graph vertices: Contains the source and sink points in the large-scale graph, as well as the starting and ending points of each edge.

[0009] Large-scale graph edge: A path connecting two adjacent vertices. The edge pointing from the starting point to the end point is called the outgoing edge of the starting point or the incoming edge of the end point.

[0010] Edge capacity: the maximum flow from the starting point to the end point of the edge.

[0011] Maximum flow of subgraph: After determining the source and sink, the maximum flow from the source to the sink of the subgraph.

[0012] Cut point: The vertex through which the flow from the source point to the sink point must pass. After deleting these points, the flow outflow from the source point will not be able to reach the sink point.

[0013] Covering graph: Perform DFS traversal on the root node of the original graph, traverse to the leaf node and then backtrack to the cut point, map the found cut point and the nodes between the cut points to a vertex in the covering graph. Each vertex in the covering graph represents a subgraph in the original graph. The covering graph is an undirected acyclic graph.

[0014] Unique path in the overlay graph: After determining the source and sink of the large-scale graph, find the subgraph from the source to the sink. The corresponding vertices of these subgraphs on the overlay graph are connected into a unique path.

[0015] Adjacent vertices in the cover graph: a pair of adjacent vertices in the cover graph between two subgraphs i and i+1 with a common cut point in a large-scale graph.

[0016] Vertex outgoing edge set: determine a vertex, take the vertex as the starting point, and the set of all edges from the starting point to the child nodes. It includes the edge from the source point to the child node and the edge from a node to its child node.

[0017] Vertex-incoming edge set: Determine a vertex, take the vertex as the end point, and the set of all edges of the parent node flowing to the end node. It includes the edge from the parent node of the sink to the sink and the edge from the parent node of a node to the node.

[0018] The present invention designs a segmentation mechanism based on the cut point (such as Figure 1 )'s large-scale graph parallel computing maximum flow acceleration method (such as Figure 2 ), mainly includes the following steps:

[0019] The first step is to use the original graph to construct the overlay graph and sub-graph. The detailed method process is as follows:

[0020]

[0021] The algorithm performs a DFS traversal from the root node of the original graph, and then backtracks to the cut points after traversing to the leaf nodes, and maps the found cut points to the overlay graph. After finding all the cut points, the corresponding connections are established in the overlay graph based on the relationships between the cut points in the original graph.

[0022] At the beginning of the algorithm, each neighbor vertex z of the current vertex x is traversed. If the neighbor vertex has not been visited, the depth value is increased by 1, and the current vertex and the neighbor vertex are pushed into the stack (lines 7 to 11).

[0023] Set the low value to INF. As the vertices are continuously traversed, the low value is updated to the vertex with the smallest depth value that can be traced back through non-father vertices (lines 12 to 20).

[0024] If the neighbor vertex z has been visited and low z ≥depth x , then x is a cut point and maps x to vertex X in the overlay graph (lines 21 to 25).

[0025] After traversing to the leaf node, backtrack upward. If the top element of the stack is not the vertex currently being visited or a neighbor vertex, the top element of the stack is popped out and the vertex that is not x in the vertex pair of the top element of the stack is added to the subgraph of the overlay graph X vertex mapping. If the top element of the stack is a source point, the current overlay graph X vertex is set as the source point of the overlay graph. If the top element of the stack is a sink point, the current overlay graph X vertex is set as the sink point of the overlay graph (lines 28 to 31).

[0026] If there is a cut point in the top element of the stack, the cut point is mapped to the overlay graph, and corresponding connections are established between the vertices of the overlay graph based on the relationship between the cut points in the original graph. If the top element of the stack is a source point, the current overlay graph U(Q) vertex is set as the source point of the overlay graph. If the top element of the stack is a sink point, the current overlay graph U(Q) vertex is set as the sink point of the overlay graph (lines 36 to 43).

[0027] The second step is to find the unique path on the coverage map for the given source and sink points. The specific implementation method is as follows:

[0028] For a given source point, first determine the subgraph it is in, and mark the vertex corresponding to the subgraph on the overlay graph as the source point of the overlay graph. Similarly, mark the vertex corresponding to the sink point on the overlay graph as the sink point of the overlay graph.

[0029] After determining the source and sink points on the coverage graph, traverse all paths from the source point until reaching the sink point. Because the coverage graph constructed by the cut point partitioning mechanism is an undirected acyclic graph, there is only one path from the source point to the sink point.

[0030] Record all the vertices on the path from the source point to the sink point on the coverage graph, and according to the direction of this unique path, find out the source point and sink point of the subgraph corresponding to each node on this path of the coverage graph.

[0031] The third step is to perform the maximum flow calculation on the subgraphs corresponding to the vertices on this path in parallel. The specific implementation process is as follows:

[0032] Submit all subgraphs corresponding to the vertices on the path from the source point to the sink point to the parallel framework GraphChi, and use the parallel framework GraphChi to calculate the maximum flow of each subgraph.

[0033] The goal is to calculate the maximum flow of each subgraph on the GraphChi platform after given the source and sink points. The formula is as follows:

[0034]

[0035] The meaning of this formula is: the maximum flow of each subgraph is equal to the source point s i To each outgoing edge point v∈δ + (s i )The sum of the maximum values ​​of the flow rates.

[0036] The restrictions are:

[0037] in,

[0038] δ - (v i )={w i ∈V i |(w i ,v i )∈E},δ + (v i )={w i ∈V i |(v i ,w i )∈E}

[0039] In order to explain this formula more clearly, the relevant terms involved in this maximum flow formula are explained as follows:

[0040] It is the set of subgraphs corresponding to the vertices on the unique path.

[0041] i is the Nth i Sub-pictures.

[0042] s i It is the Nth i The source point of the subgraph.

[0043] t i It is the Nth i The sink of the subgraph.

[0044] c e Represents the capacity of each edge.

[0045] f e Indicates the flow rate on each edge

[0046] f(v i ,w i ) is from vertex v i To the vertex w i of traffic.

[0047] f(w i ,v i ) is from the top w i Point to vertex v i of traffic.

[0048] f(s i ,v i ) is from vertex s i To the vertex v i of traffic.

[0049] δ + (s i ) represents the vertex s of the i subgraph i The set of child nodes of outgoing edges.

[0050] δ - (s i ) represents the vertex s of the i subgraph i The set of child nodes of the incoming edge.

[0051] E i Represents the set of all edges in subgraph i.

[0052] Represents the vertex v of the i subgraph i The value of is except the source point s i , meeting point i The set V of all points other than i .

[0053] δ - (v i )={w i ∈V i (w i ,v i )∈E i} represents the vertex v of the i subgraph i The set of all incoming edges. Each vertex w in the set i All belong to the vertex set V i , and vertex w i With vertex v i The edges between them belong to the edge set E i .

[0054] δ + (v i )={w i ∈V i (v i ,w i )∈E i} represents the vertex v of the i subgraph i The set of all outgoing edges. Each vertex w in the set i All belong to the vertex set V i , and vertex w i With vertex v i The edges between them belong to the edge set E i .

[0055] The meanings of restrictions (a), (b) and (c) are:

[0056] (a) represents the source point s of the i subgraph i To each outgoing vertex v i ∈δ + (s i ) is equal to the sum of the maximum flow of each inbound point v i ∈δ - (t i ) to the meeting point t i The sum of the maximum flow rates.

[0057] (b) represents the flow f of each edge e in subgraph i e It is non-negative and less than the capacity of the edge.

[0058] (c) represents the vertex v of the i subgraph i The sum of the flows to all outgoing edges is equal to the flows from all incoming edges to vertex v i The sum of the traffic.

[0059] The above formula obtains the maximum flow of the i-th subgraph on the coverage graph path. The solution to the problem is to solve the maximum flow of each subgraph on the coverage graph path in parallel. The minimum value of the maximum flow of all subgraphs on the path corresponds to the maximum flow value of the original large-scale graph, that is:

[0060]

[0061] In order to explain this formula more clearly, the relevant terms involved in this maximum flow formula are explained as follows:

[0062] {N1,N2,N3,.....N k} represents the set of vertices on the unique path of the covering graph

[0063] i represents a point on the coverage graph path, which is a subgraph of the original large-scale graph.

[0064] s i Represents the source point in the i subgraph

[0065] t i Represents the sink in the i subgraph

[0066] The meaning of this formula: Formula ① means to find the maximum flow of all subgraphs on the unique path, and take the minimum value of the maximum flow of the subgraphs. This value is the maximum flow of the entire coverage graph, that is, the maximum flow of the original large-scale graph.

[0067] in,

[0068]

[0069] (d) means to find N1→N2→N3→.....→N k The maximum flow of each subgraph on the path is taken, and the minimum value is taken. This minimum value is equivalent to the maximum value of the sum of the flow of each branch of the original large-scale graph. Let subgraph i∈{N1,N2,N3,.....N k The maximum flow is the smallest of all maximum flows. In this case, (d) in the objective formula is expressed as the minimum value of the maximum flow on the path for the i-th subgraph.

[0070] The solution to the original problem is to find the value of the maximum flow of the entire large-scale graph, while the problem goal of the cut point partitioning mechanism is to use the GraphChi platform to parallelly calculate the maximum flow values ​​of all subgraphs on a given path, and find the maximum flow on this path, that is, the minimum value (d) of the maximum flow of each subgraph, to obtain the maximum flow value of the original large scale.

[0071] The maximum flow solution algorithm on the coverage graph, the detailed algorithm process is as follows:

[0072]

[0073] Beneficial effects:

[0074] 1. By using the present invention, the parallel computing capabilities of the GraphChi platform can be fully utilized, solving bottleneck problems such as high complexity and maintenance difficulties of distributed computing methods, low computing efficiency of traditional serial methods, and inability to fully utilize computing resources.

[0075] 2. The present invention utilizes the cut point segmentation mechanism to construct the original large-scale graph coverage map. After determining the corresponding unique path from the source point to the sink point on the coverage map, all subgraphs corresponding to the vertices on the path are submitted to the GraphChi platform for parallel maximum flow calculation, which ensures the independence of the maximum flow calculation of each subgraph and reduces the number of communications. By parallelizing the maximum flow calculation and then integrating the results, the computational efficiency and correctness are guaranteed, which can meet the needs of developing methods for quickly solving the maximum flow of large-scale graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 Constructing a cover map of the original graph using the cut point partitioning mechanism

[0077] Figure 2 Based on the GraphChi framework, the maximum flow is solved in parallel for the subgraphs corresponding to the vertices on the unique path of the coverage graph.

[0078] Figure 3 Comparison chart of execution speed of serial program and parallel program under the same data. DETAILED DESCRIPTION

[0079] Embodiment 1:

[0080] by Figure 1 , Figure 2 and Figure 3 (in, Figure 1 The solid circles in the left figure represent vertices, letters represent vertex numbers, dashed lines represent edges between two points, shaded vertices s represent sinks, and shaded vertices t represent sinks. Figure 1 In the figure on the right, the dotted circles represent the vertices of the cover graph, the capital letters represent the vertex labels, and the solid lines represent the edges between two cover graph vertices; Figure 2 The shadow vertices form the only path from the source to the sink of the coverage graph, and the graph next to each vertex is the subgraph corresponding to the vertex; Figure 3 The rectangular points represent the parallel time-node scale relationship diagram. Figure 3 Taking the relationship diagram of serial line time and node scale as an example, the implementation process of the present invention is described in detail, and its specific implementation method is as follows:

[0081] First, use the cut point partitioning mechanism to construct the original graph ( Figure 1 Left) coverage map ( Figure 1 Right), and use the source and sink points of the original graph to find a unique path on the overlay graph. According to the direction of this path, the source and sink points of the subgraph corresponding to the vertices on this path are found and marked.

[0082] Bundle( Figure 2 )The subgraph corresponding to the vertices on the unique path is submitted to the GraphChi platform for parallel maximum flow calculation, and then the maximum flow value on the path is found.

[0083] Figure 3 The figure shows the difference between the execution time of calculating the maximum flow on different data sets between the present invention and the conventional serial method. The data comes from the New York City data in TIGER / Line. The horizontal axis in the figure represents the number of vertices selected from the original large-scale graph, and the vertical axis represents the program execution time. As the data set gradually increases from the data with 321270 vertices, the execution time of the parallel calculation of the maximum flow method on the GraphChi framework increases slowly, while the execution time of the traditional serial method increases sharply. This is because the overlay graph and subgraph on data sets of different sizes are generated only once, and the necessary subgraphs are calculated in parallel. Therefore, the execution time of the maximum flow can be significantly shortened when performing maximum flow calculation on large-scale data sets. This shows that the larger the data scale, the more obvious the acceleration effect of the present invention.

[0084] We compared the traditional serial method with the method of solving the maximum flow of subgraphs in parallel using the GraphChi framework based on the cutpoint partitioning mechanism. Based on the comparison of the execution time of the two methods on the same data, we concluded the acceleration effect of the method of parallel computing the maximum flow of subgraphs using the cutpoint partitioning mechanism. In the same computing environment, the larger the data set and the more evenly distributed the number of subgraph nodes, the shorter the execution time of the accelerated method of parallel computing the maximum flow of subgraphs using the cutpoint partitioning mechanism on the GraphChi platform, and the better the acceleration effect of the method.

Claims

1. An acceleration method for large-scale graph parallel computing of maximum flow based on a cut point partitioning mechanism, characterized by: Processing large-scale graphs on a single machine, the problem of solving the maximum flow of large-scale graphs is transformed into the problem of solving the maximum flow of multiple subgraphs in parallel. The overlay graph is constructed based on the cut point partitioning mechanism, and the subgraphs corresponding to the vertices on the unique path of the overlay graph are quickly solved under the parallel GraphChi framework; The original large-scale map construction coverage map mainly includes the following steps: At the beginning of the algorithm, traverse the current vertex Every neighbor vertex of , if the neighbor vertex has not been visited, then The value is increased by 1, and the current vertex and neighboring vertices are pushed into the stack; set up Value is , as the vertices are continuously traversed, The value is updated to be traceable through non-parent vertices The vertex with the smallest value; If the neighbor vertex Has been visited and ,but As the cutting point, Mapping to the vertices in the overlay ; After traversing to the leaf node, backtrack upward. If the top element of the stack is not the vertex currently being visited or the neighbor vertex, the top element of the stack is popped out and the vertex of the top element of the stack is not The vertices are added to the coverage graph In the subgraph of the vertex map, if the top element of the stack is the source point, the current overlay map The vertex is set as the source point of the overlay graph. If the top element of the stack is a sink, the current overlay graph is The vertices are set as sinks of the overlay graph; If there is a cut point in the top element of the stack, the cut point is mapped to the overlay graph, and corresponding connections are established between the vertices in the overlay graph based on the relationship between the cut points in the original graph. If the top element of the stack is a source point, the current overlay graph is The vertex is set as the source point of the overlay graph. If the top element of the stack is a sink, the current overlay graph is The vertices are set as sinks of the overlay graph; Given a source point and a sink point, find the only path on the coverage map. The specific implementation method is as follows: For a given source point, first determine the subgraph it is in, and mark the vertex corresponding to the subgraph on the overlay graph as the source point of the overlay graph. Similarly, mark the vertex corresponding to the sink point on the overlay graph as the sink point of the overlay graph. After determining the source and sink points on the coverage graph, traverse all paths from the source point until reaching the sink point. Because the coverage graph constructed by the cut point partitioning mechanism is an undirected acyclic graph, there is only one path from the source point to the sink point. Record all the vertices on the path from the source point to the sink point on the coverage graph, and find the source and sink points of the subgraph corresponding to each node on this path on the coverage graph according to the direction of this unique path; The unique path of the coverage graph can be transformed into a multi-subgraph parallel maximum flow problem, which mainly includes the following steps: The objective formula of the problem is expressed as: The meaning of this formula is: the maximum flow of each subgraph is equal to the sum of the maximum values ​​of the flow from the source point to each outgoing edge point; The restrictions are: in, ， is the set of subgraphs corresponding to the vertices on the unique path It is Subgraph It is The source of the subgraph It is The sink of the subgraph Represents the capacity of each edge Indicates the flow on each edge From the vertex To the top Traffic It's from the top Point to Vertex Traffic From the vertex To the top Traffic express Subgraph Vertex The set of child nodes of outgoing edges express Subgraph Vertex The set of child nodes of the incoming edge express The set of all edges in the subgraph Represents the vertices of the subgraph The value of is except the source point , Huidian The set of all points except express Subgraph Vertex The set of all incoming edges, each vertex in the set All belong to the vertex set , and the vertex With Vertex The edges between them belong to the edge set express Subgraph Vertex The set of all outgoing edges, each vertex in the set All belong to the vertex set , and the vertex With Vertex The edges between them belong to the edge set b. The above formula obtains the maximum flow of the subgraph on the coverage graph path. The solution to the problem is to solve the maximum flow of each subgraph on the coverage graph path in parallel. The minimum value of the maximum flow of all subgraphs on the path corresponds to the maximum flow value of the original large-scale graph, that is: Represents the set of vertices on a unique path in the covering graph Represents a point on the overlay graph path, which is a subgraph of the original large-scale graph express Source points in subgraphs express Sink points in subgraphs c. Formula ② means to find the maximum flow of all subgraphs on the unique path, and take the minimum value of the maximum flow of the subgraphs. This value is the maximum flow of the entire coverage graph, that is, the maximum flow of the original large-scale graph.

Images