K-truss decomposition method based on Quegel distributed graph computing system
By using a vertex-centric distributed k-truss decomposition method and the Quegel distributed graph computing system, the inefficiency of traditional methods in processing large network graphs is solved, achieving efficient multi-machine node decomposition and reducing memory requirements.
Patent Information
- Application Number
- CN202110814231.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-19
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-07-19
AI Technical Summary
Traditional k-truss serial methods are inefficient when processing large network graphs and require a lot of memory, which cannot meet the processing needs of large graphs.
The traditional k-truss serial method is modified into a vertex-centric distributed k-truss decomposition method. The Quegel distributed graph computing system is used to decompose the large graph through multiple machine nodes, and MPI AllReduce is used for message communication and synchronization to realize k-truss decomposition on a distributed system.
It improves the decomposition efficiency of large network graphs, reduces the demand for single-machine memory, and can effectively process large graph data.
Smart Images

Figure CN113590887B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a K-truss decomposition method based on a Quegel distributed graph computing system. Background Art
[0002] With the development of society, there are many large graphs in social life: for example, the social networks of users in social software (QQ, WeChat, Facebook), the transportation networks between provinces, protein networks, etc. The processing of these graphs is of great value in social applications, such as helping the marketing of goods. It can expand the influence of goods or prevent the spread of some unfavorable opinions, or it can detect and find some specific topological structures from protein networks to complete some research. There are often quite a lot of nodes and edges in these graphs. For these large graphs, the traditional k-truss serial decomposition method can no longer meet the processing requirements. In this case, on the one hand, many scholars have made breakthroughs in the direction of k-truss parallel methods, and on the other hand, they use distributed computing systems for processing. Summary of the invention
[0003] In order to solve the above technical problems existing in the prior art, the present invention proposes a K-truss decomposition method based on the Quegel distributed graph computing system. For large network graphs, the traditional k-truss serial method is modified into a distributed k-truss decomposition method on a vertex-centered graph. The Quegel distributed graph computing system is used to modify the source code of the system to realize the k-truss decomposition of large graphs on the distributed system. The specific technical solution is as follows:
[0004] The K-truss decomposition method based on the Quegel distributed graph computing system includes the following steps:
[0005] Step 1, start the run function in the Quegel distributed graph computing system, load the graph G = (V, E), where V is the vertex set of graph G, E is the edge set of graph G, read the file storing the graph line by line, convert it into a string, and then call the custom UDF function. The custom UDF function converts a line of character data in the file into vertices and adjacency lists, and stores the edge set with vertices as endpoints in global_Edge_Map;
[0006] Step 2: After the graph is loaded, reset the timer through the decomposition function, create a new query task for decomposition, initialize the query task and add it to the query task queue, and then clear the communication and calculation volume of the query task queue;
[0007] Step 3, execute the query task queue, activate all vertices, and perform K-truss decomposition on all activated vertices by calling the decomposition function;
[0008] Step 4: Count and determine the number of edges in global_Edge_Map. If it is empty, determine that the decomposition is complete and exit the process. Then print the calculation amount, communication amount and decomposition time information.
[0009] Furthermore, the step 1 specifically includes:
[0010] Step 1.1: After the Quegel distributed graph computing system is running, the loading graph function is called first. The function first opens the file specified by the input path, and then reads the file line by line. Each time a line of input data is read, the custom UDF function is called.
[0011] Step 1.2, by calling the custom UDF function, take the first character of each line as the vertex, skip the number of neighbors, read and store the adjacency list, and store the edge set corresponding to the edge to the vertex in global_Edge_Map.
[0012] Furthermore, when the query task queue is not cleared or the number of edges in global_Edge_Map is not empty, the Quegel distributed graph computing system will continue to repeat the following process:
[0013] a. The number of supersteps increases, all vertices are activated, and the current appropriate global_k value is sought;
[0014] b. Execute the custom UDF function on all activated vertices to perform logical calculations;
[0015] c. Message communication and synchronization. MPI AllReduce is used to count the number of edges in the global_Edge_Map. If the global_Edge_Map of a worker is empty, the decomposition is completed.
[0016] Furthermore, the decomposition function is responsible for judging and processing the number of supersteps of the tasks executed by the activated vertices.
[0017] Furthermore, the number of supersteps is specifically:
[0018] Superstep 1: traverse the adjacency list nbs of the current vertex v, and send a message to all neighbors in the adjacency list of v. The message content is v's id and v's adjacency list nbs; v is a vertex in the vertex set V;
[0019] Superstep 2: Traverse the received message set messages. For each message received, assume that the source vertex of the message is u, find the position of (v, u) in the e set of the current vertex v, then find the intersection between the adjacency list of v and the adjacency list of u, and then save the intersection to the v3 set of the edge (v, u), and set the support value of the edge e=(v, u) to the number of intersections, and modify the support value of the edge (v, u) stored in the global_Edge_Map; where (v, u) in the e set is an edge in E, and the two end vertices of the edge are u and v; support is the number of triangles included in the edge e in the graph G or the number of triangles that the edge e can form with other edges in the graph G; messages is the set of messages sent by a vertex to other vertices;
[0020] Superstep 3 and later: traverse all edges of the current vertex v. If the support value of an edge e(v, u) is less than or equal to global_k and e has not been decomposed, delete e from the global_Edge_Map, set the trussness value of e to the support value of e plus 2, set e to be decomposed, and print relevant information. Then, traverse the v3 set of e. Assuming that the vertex stored in the v3 set is w, call the find_epos function to find the position of (v, w) in the edge set of the current v. Then, if the support value of (v, w) is greater than global_k, subtract one and update the support value of the corresponding edge (v, w) in the global_Edge_Map. Otherwise, if it is not greater than global_k, keep it unchanged. The trussness value of e is the k value of the maximum k-turss of the edge e after decomposition.
[0021] Advantages of the present invention:
[0022] The present invention aims at large network graphs, modifies the traditional k-truss serial method into a distributed k-truss decomposition method on vertex-centered graphs, and uses the Quegel distributed graph computing system to realize k-truss decomposition processing of large graphs on distributed systems, so that multiple ordinary machine nodes can be used to perform k-truss decomposition on large graphs, which is helpful for the analysis and processing of large graphs in current life, and solves the problem that most of the current technical solutions are k-truss decomposition processing of graphs on one machine, which requires very high machine configuration and consumes a lot of memory when processing large graphs. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 A schematic diagram of an embodiment using the conventional K-truss decomposition method;
[0024] Figure 2 A schematic diagram of the K-truss decomposition process of the Quegel distributed graph computing system of the present invention;
[0025] Figure 3 A schematic diagram of determining and processing the number of supersteps of a task executed by the KTrussdecompositon() function of the present invention on the current vertex. DETAILED DESCRIPTION
[0026] In order to make the purpose, technical scheme and technical effect of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0027] The existing traditional k-truss decomposition method is mainly divided into two steps. The first step is to calculate the support values of all edges in the graph through triangle counting method or bitmap, and then store them. The second step is to find the edge with the smallest support, remove e, and update the support values of the other two edges that can form a triangle with edge e, until all edges are removed, and then the trussness(e) value of each edge e is support(e)+2.
[0028] Specifically, Figure 1 As shown, first count the triangles of the graph and find the support of all edges. Value, as shown in the figure, support(AB) is equal to 2, and then traverse all edges to find the edge with the smallest support value. There are two edges with the smallest support values in the figure: CD and DE. Suppose CD is found first, k is equal to support(CD) is equal to 1, remove CD, and then find other edges CE and DE that can form a triangle with CD. Because support(DE) is equal to k, support(DE) remains unchanged, and support(CE) is equal to 3, which is greater than k, so support(CE) minus 1 becomes 2, and then continue to search. It is found that the current support value of DE is the smallest, so DE is removed. It is found that after CD is removed, there is no edge that can form a triangle with DE, so continue to search. Suppose CE is found first, k is equal to support(CE) is equal to 2, and find the edges that can form a triangle with CE, AB, AC, BE, AE, the support values of these edges are not greater than k, so no changes are made, and continue to search. This process continues. Finally, after each edge is removed, trussness(e) is equal to support(e) plus 2, for example, trussness(CD) is equal to 1 plus 2, which is equal to 3.
[0029] The existing traditional k-truss decomposition method can only be used on one machine. When encountering large graphs, not only is the overall decomposition efficiency low, but the storage of large graphs and the data structures generated during the decomposition process all require a large amount of memory, which is a test of the machine memory. The general machine memory is not enough to process large graphs.
[0030] like Figure 2 As shown, the K-truss decomposition method based on the Quegel distributed graph computing system of the present invention is specifically as follows:
[0031] First, start with the run function, which is used to start the job of the Quegel distributed graph computing system:
[0032] First, call the loading graph function and define the loading graph function in the ol / WorkOl_auto.h file: load_graph_from_txt() function. This function first opens the file specified by the input path, and then reads the file line by line. Each time a line of input data is read, the custom UDF function toVertex function in the Worker class is called.
[0033] The toVertex function converts a line of character data in the input file into vertices and an adjacency list, and stores the edge set with the vertices as endpoints in the global_Edge_Map. Specifically, the first character of each line is the vertex, the number of neighbors is skipped, the adjacency list is read and stored, and the edge set from the corresponding edge to the vertex is stored in the global_Edge_Map.
[0034] After the loading of graph G = (V, E) is completed, the k-turss decomposition of graph G = (V, E) is started, where V is the vertex set of graph G, E is the edge set of graph G, k-turss is the cohesive subgraph of graph G, and each edge in k-turss is contained in at least k-2 triangles.
[0035] The k-truss decomposition function defined in this embodiment is named KTruss_Decompositon_t, which is defined in the ol / WorkOl_auto.h file, that is, calling the KTruss_Decompositon_t function to perform k-truss decomposition on the graph G=(V, E).
[0036] The KTruss_Decomposition_t function will first reset the timer and then create a new query task as the decomposition task. Because the Quegel distributed graph computing system is query-centric, after the task is initialized, it is added to the query task queue, clearing the communication and computational traffic, and then activating all vertices to execute the query task queue.
[0037] When the query task queue is not cleared or the global decomposition is judged as not finished, the following process will be repeated:
[0038] a. The number of supersteps increases, all vertices are activated, and find_global_k is called to find the current appropriate global_k value.
[0039] b. Execute the UDF function compute in the custom vertex class for all activated vertices to perform logical calculations. In this embodiment, the compute function is defined as the KTrussdecompositon() function;
[0040] c. Message communication and synchronization, use MPI AllReduce to count the number of edges in the global_Edge_Map. If the global_Edge_Map of a worker is empty, the decomposition flag position is decomposition completed, and the process exits.
[0041] After exiting the process, the computation amount, communication amount, and decomposition time information are printed.
[0042] The KTrussdecompositon() function is responsible for the logical calculation of each vertex of the decomposition task. The function determines the number of supersteps of the task currently executed by the vertex and performs different processing at different supersteps. The specific process is as follows: Figure 3 As shown, including:
[0043] Superstep 1: Traverse the adjacency list nbs of the current vertex v, and send a message to the neighbors in all adjacency lists of v. The message content is v's id and v's adjacency list nbs; v is a vertex in the vertex set V.
[0044] Superstep 2: Traverse the received message set messages. For each message received, assume that the source vertex of the message is u, find the position of (v, u) in the e set of the current vertex v, then find the intersection between the adjacency list of v and the adjacency list of u, and save the intersection to the v3 set of the edge (v, u), and set the support value of the edge e=(v, u) to the number of intersections, and modify the support value of the edge (v, u) stored in the global_Edge_Map; where (v, u) in the e set is an edge in E, and the two end vertices of the edge are u and v; support is the number of triangles included in the edge e in the graph G or the number of triangles that the edge e can form with other edges in the graph G; messages is the set of messages sent by a vertex to other vertices.
[0045] Superstep 3 and later: traverse all edges of the current vertex v. If the support value of an edge e (v, u) is less than or equal to global_k and e has not been decomposed, then delete e from the global_Edge_Map and set the trussness value of e to the support value of e plus 2. Set e to be decomposed and print relevant information. Then, traverse the v3 set of e. Assuming that the vertex stored in the v3 set is w, call the find_epos function to find the position of (v, w) in the edge set of the current v. Then, if the support value of (v, w) is greater than global_k, subtract one and update the support value of the corresponding edge (v, w) in the global_Edge_Map. Otherwise, if it is not greater than global_k, it remains unchanged. The trussness value of e is the k value of the maximum k-turss of the edge e after decomposition.
[0046] The above is only a preferred implementation case of the present invention and does not limit the present invention in any form. Although the implementation process of the present invention is described in detail above, for those familiar with the art, they can still modify the technical solutions recorded in the above examples, or replace some of the technical features therein with equivalents. All modifications, equivalent replacements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. The K-truss decomposition method based on the Quegel distributed graph computing system is characterized by: The steps include: Step 1, start the run function in the Quegel distributed graph computing system, load the graph G = (V, E), where V is the vertex set of graph G, E is the edge set of graph G, read the file storing the graph line by line, convert it into a string, and then call the custom UDF function. The custom UDF function converts a line of character data in the file into vertices and adjacency lists, and stores the edge set with vertices as endpoints in global_Edge_Map; Step 2: After the graph is loaded, reset the timer through the decomposition function, create a new query task for decomposition, initialize the query task and add it to the query task queue, and then clear the communication and calculation volume of the query task queue; Step 3, execute the query task queue, activate all vertices, and perform K-truss decomposition on all activated vertices by calling the decomposition function; Step 4: Count and determine the number of edges in global_Edge_Map. If it is empty, determine that the decomposition is complete and exit the process. Then print the calculation amount, communication amount and decomposition time information.
2. The K-truss decomposition method based on the Quegel distributed graph computing system according to claim 1, characterized in that: The step 1 specifically includes: Step 1.1: After the Quegel distributed graph computing system is running, the loading graph function is called first. The function first opens the file specified by the input path, and then reads the file line by line. Each time a line of input data is read, the custom UDF function is called. Step 1.2, by calling the custom UDF function, take the first character of each line as the vertex, skip the number of neighbors, read and store the adjacency list, and store the edge set corresponding to the edge to the vertex in global_Edge_Map.
3. The K-truss decomposition method based on the Quegel distributed graph computing system according to claim 1, characterized in that: When the query task queue is not cleared or the number of edges in global_Edge_Map is not empty, the Quegel distributed graph computing system will continue to repeat the following process: a. The number of supersteps increases, all vertices are activated, and the current appropriate global_k value is sought; b. Execute the custom UDF function on all activated vertices to perform logical calculations; c. Message communication and synchronization. MPI AllReduce is used to count the number of edges in the global_Edge_Map. If the global_Edge_Map of a worker is empty, the decomposition is completed.
4. The K-truss decomposition method based on the Quegel distributed graph computing system according to claim 1, characterized in that: The decomposition function is responsible for judging and processing the number of supersteps of the tasks executed by the activated vertices.
5. The K-truss decomposition method based on the Quegel distributed graph computing system according to claim 4, characterized in that: The number of supersteps is specifically: Superstep 1: traverse the adjacency list nbs of the current vertex v, and send a message to all neighbors in the adjacency list of v. The message content is v's id and v's adjacency list nbs; v is a vertex in the vertex set V; Superstep 2: Traverse the received message set messages. For each message received, assume that the source vertex of the message is u, find the position of (v, u) in the e set of the current vertex v, then find the intersection between the adjacency list of v and the adjacency list of u, and then save the intersection to the v3 set of the edge (v, u), and set the support value of the edge e=(v, u) to the number of intersections, and modify the support value of the edge (v, u) stored in the global_Edge_Map; where (v, u) in the e set is an edge in E, and the two end vertices of the edge are u and v; support is the number of triangles included in the edge e in the graph G or the number of triangles that the edge e can form with other edges in the graph G; messages is the set of messages sent by a vertex to other vertices; Superstep 3 and later: traverse all edges of the current vertex v. If the support value of an edge e(v, u) is less than or equal to global_k and e has not been decomposed, delete e from the global_Edge_Map, set the trussness value of e to the support value of e plus 2, set e to be decomposed, and print relevant information. Then, traverse the v3 set of e. Assuming that the vertex stored in the v3 set is w, find the position of (v, w) in the edge set of the current v, and then determine if the support value of (v, w) is greater than global_k, then subtract one and update the support value of the corresponding edge (v, w) in the global_Edge_Map. Otherwise, if it is not greater than global_k, it remains unchanged. The trussness value of e is the k value of the maximum k-turss of the edge e after decomposition.
