A community detection evaluation method based on graph algebra

Abstract

The present application relates to the technical field of graph data mining, and particularly relates to a community detection evaluation method based on graph algebra. The method adopts a modularity calculation method in the form of graph algebra, and can construct a modularity matrix in a lock-free parallel manner, and is superior to a traditional method in time complexity and calculation efficiency. Through verification of real graph data sets and divided data sets, the method has great advantages compared with the traditional method.

Description

Technical Field

[0001] This invention relates to the field of graph data mining technology, and in particular to a community detection and evaluation method based on graph algebra. Background Technology

[0002] With the rapid development of technology and the high-speed advancement of big data and networks, humanity has entered an era of internet "information explosion," leading to an unprecedented increase in the storage and management of massive amounts of data, particularly graph data. A graph consists of nodes and edges representing the relationships between them. In real life, nodes can represent entities, and edges can represent relationships or attributes between entities. This abstract structure can be well integrated with the many concrete systems needed in life. In graph networks, tightly coupled subgraphs are called communities. Community detection algorithms can find all communities in the current network, but there are many types of communities, including cliques, k-cores, k-truss, k-plexes, and their derivatives. The complexity of community detection algorithms and the need for a numerical quality assessment of the community divisions obtained through community detection are urgent problems to be solved. The paper "Modularity and community structure in networks" by Newman et al. proposed a modularity index, providing a clue to this problem.

[0003] However, the modularity calculation in the paper is mainly sequential, based on the following formula:

[0004]

[0005] Where A is the adjacency matrix of the graph, A ij k represents the weight of the edge between nodes i and j (1 in an unweighted graph). i c represents the sum of the weights of all edges connected to node i (or the degree of node i in an unweighted graph). i c represents the community where node i is located. j δ(c) represents the community where node j is located. i ,c j The function is a delta function with respect to the community (returning 1 if the inputs are equal, and 0 otherwise). Many high-performance open-source network analysis libraries (such as NetworKit and igraph) currently use this method to calculate modularity.

[0006] However, this algorithm requires traversing the adjacent edges of each node. Since each edge has two nodes, all edges are visited twice, resulting in an overall time complexity of O(2E). This is relatively high, with a long theoretical computation time. It can only handle modularity solutions on smaller datasets and is slow on larger datasets. Furthermore, current mainstream open-source libraries implement it serially or in partial parallelism, resulting in low program efficiency and failing to fully utilize the advantages of current mainstream multi-core processors. Summary of the Invention

[0007] The purpose of this invention is to provide a community detection and evaluation method based on graph algebra. This method uses a modularity calculation method in the form of graph algebra and can construct the modularity matrix in parallel without locking. It is superior to traditional methods in terms of time complexity and computational efficiency. After verification with real graph datasets and partitioned datasets, this method has significant advantages over traditional methods.

[0008] The objective of this invention is achieved through the following technical solution:

[0009] A community detection and evaluation method based on graph algebra, for a given graph G and partitioning data P, the method includes:

[0010] Step 1: Remap the partitioned data P to obtain new partitioned data P_new, where the partition IDs of the new partitioned data P_new are consecutive;

[0011] Step 2: Based on the edge set data of graph G and the new partition data P_new obtained in step 1, construct the modularity matrix E in parallel without locking.

[0012] Step 3: Perform graph algebra calculations on the modularity matrix E obtained in Step 2 to obtain the modularity M, which is the community detection and evaluation result based on graph algebra.

[0013] In step one, the data structure of the partition data P is a one-dimensional array. The index of the one-dimensional array is the node vrt_id in graph G, and the value at the index of the node vrt_id in graph G is the partition prt_id corresponding to the current node. The partition data P is essentially a mapping from vrt_id to prt_id. The prt_id in the partition data P can be non-contiguous.

[0014] In step one, the method for remapping the partitioned data P to obtain the new partitioned data P_new is as follows:

[0015] (1) Construct a mapping between the old partition ID and the new partition ID, and assign the maximum partition max_id to 0;

[0016] (2) Start accessing prt_id in the old partitioned data;

[0017] If the current prt_id has not been remapped, then the current prt_id is mapped to max_id, and max_id is incremented by 1;

[0018] If the current prt_id has been remapped, then the current prt_id will not be remapped.

[0019] (3) Return to step (2) until all nodes are traversed and the new partition data P_new is obtained.

[0020] In step two, the data structure in graph G is an edge set array, a two-dimensional array, or a structure that can iterate over the edges.

[0021] In step two, the method for constructing the modularity matrix E is as follows:

[0022] (1) Initialize the submatrix, with the number of submatrices equal to the maximum number of threads;

[0023] (2) All edges in graph G are assigned, and different threads process the assigned edges in parallel;

[0024] (3) Traverse the edges assigned to the current thread;

[0025] (4) For edge (i,j), according to the new partition data, the partition IDs corresponding to i and j are c(i) and c(j). The elements e(c(i), c(j)) and e(c(i), c(j)) corresponding to the current thread submatrix are all added with edge weight values ​​(add 1 for unweighted graphs). i and j are the two node IDs of the edge.

[0026] (5) Return to step (3) until all edges have been processed;

[0027] (6) Iterate through all non-zero elements e(i,j) of the obtained submatrixes, and add e(i,j) to the corresponding modularity matrix element E(i,j);

[0028] (7) Divide all non-zero elements of the matrix by twice the sum of the edge weights to obtain the modularity matrix E;

[0029] In step three, performing the corresponding graph algebra calculation on the modularity matrix E means first calculating the trace Tr(E) of the modularity matrix E, and then calculating the modulus ||E| of the square of the modularity matrix E. 2 Finally, we obtain the modularity M = Tr(E) - ||E 2 ||.

[0030] Beneficial effects

[0031] This method employs a graph algebraic approach to calculate modularity and can construct the modularity matrix in parallel without locking. It outperforms traditional methods in terms of time complexity and computational efficiency. Validated on real graph datasets and partitioned datasets, this method demonstrates significant advantages over traditional methods. Attached Figure Description

[0032] Figure 1 This is a sample undirected unweighted graph and its original partition.

[0033] Figure 2 It is an undirected, unweighted graph that has been repartitioned from the ID mapping.

[0034] Figure 3 A schematic diagram for constructing the modularity matrix of an unweighted graph in a multithreaded manner;

[0035] Figure 4 This is a sample undirected weighted graph and its original partitioning;

[0036] Figure 5 This is an undirected weighted graph with redefined ID mappings;

[0037] Figure 6 This is a schematic diagram of constructing the modularity matrix of a weighted graph using multithreading. Detailed Implementation

[0038] The technical solution of the present invention will now be described more clearly and in detail with reference to the accompanying drawings. For the given example, to simplify the explanation, it is assumed that the current number of program threads is 3.

[0039] Example 1

[0040] Given an undirected, unweighted graph G and partitioned data P, such as Figure 1 As shown below, the specific implementation of this method in undirected and unweighted graphs will be described.

[0041] Step 1: Remap the partition data P = [null, 1, 1, 1, 6, 6, 7, 7, 7, 7] to obtain new partition data P_new, where the partition IDs of the new partition data are consecutive. The specific steps are as follows:

[0042] (1) Initialize the unordered mapping container mapp to store the mapping from the old partition ID to the new partition ID, initialize the new partition data P_new, and set the maximum partition max_id to 0;

[0043] (2) Iterate through nodes 1 to 9;

[0044] (3) If the current prt_id has not been remapped, then the current prt_id is mapped to max_id, and max_id is incremented by 1;

[0045] If the current prt_id has been remapped, then the current prt_id will not be remapped.

[0046] (4) Obtain the continuous partition IDs corresponding to the current node based on the partition ID mapping data, and use them as new partition data;

[0047] (5) Return to step (2) until all nodes have been traversed;

[0048] (6) Return the newly partitioned data P_new = [null, 0, 0, 0, 1, 1, 2, 2, 2, 2].

[0049] An undirected, unweighted graph after repartitioning the ID mapping is as follows: Figure 2 As shown;

[0050] Step 2: Based on the edge set data E of the graph and the partition data P_new obtained in Step 1, construct the modularity matrix E in parallel without locking. See the schematic diagram below. Figure 3 ;

[0051] (1). Initialize the submatrix array m[], with a size equal to the number of threads, 3;

[0052] (2). The edge set E of the graph is allocated, and different threads process the allocated edges in parallel;

[0053] (3) Traverse the edges assigned to the current thread p_id;

[0054] (4). For edge e(i,j), obtain the partition c corresponding to points i and j according to the new partition data P_new. i ,c j ;

[0055] (5). The element m(c) corresponding to the current thread's submatrix i ,c j ), m(c j ,c i Add 1 to each;

[0056] (6). Return to step (3) until all edges have been processed;

[0057] (7). For all non-zero elements of the obtained submatrices, add m[p_id](i,j) to the corresponding modularity matrix E(i,j);

[0058] (8) Let the total weight W be equal to the size of the edge set of graph G;

[0059] (9). E multiplied by

[0060] (10). Return the modularity matrix.

[0061] Step 3: Calculate the modularity M using graph algebra based on the obtained modularity matrix. The specific steps are as follows:

[0062] (1). Calculate the trace Tr(E) of the modularity matrix E = 5 / 6;

[0063] (2). Calculate the modulus ||E|| of the squared modularity matrix E. 2 ||=0.40625;

[0064] (3). The modularity M is calculated as Tr(E) - ||E 2||=0.427083;

[0065] (4). The module degree returned is M = 0.427083.

[0066] Example 2

[0067] Given an undirected weighted graph G and a partition P, if... Figure 4 As shown below, the specific implementation of this method in undirected and unweighted graphs will be described.

[0068] Step 1: Remap the partition data P = [null, 1, 1, 1, 6, 6, 7, 7, 7, 7] to obtain new partition data P_new, where the partition IDs of the new partition data are consecutive. The specific steps are as follows:

[0069] (1). Initialize the unordered mapping container mapp to store the mapping from the old partition ID to the new partition ID, initialize the new partition data P_new, and the maximum partition max_id = 0;

[0070] (2). Iterate through nodes 1 to 9;

[0071] (3). If the current prt_id has not been remapped, then the current prt_id is mapped to max_id, and max_id is incremented by 1;

[0072] If the current prt_id has been remapped, then the current prt_id will not be remapped.

[0073] (4). Obtain the continuous partition IDs corresponding to the current node based on the partition ID mapping data, and use these as the new partition data;

[0074] (5). Return to step (2) until all nodes have been traversed;

[0075] (6). Return the newly partitioned data P_new = [null, 0, 0, 0, 1, 1, 2, 2, 2, 2].

[0076] An undirected weighted graph after repartitioning the ID mapping is as follows: Figure 5 As shown;

[0077] Step 2: Based on the edge set data E of the graph and the partition data P_new obtained in Step 1, construct the modularity matrix E in parallel without locking. See the schematic diagram below. Figure 6 ;

[0078] (1). Initialize the submatrix array m[] to be a matrix of all zeros and the weight array w[] (with elements of 0), both with a size of 3 (the number of threads).

[0079] (2). The edge set E of the graph is allocated, and different threads process the allocated edges in parallel;

[0080] (3) Iterate through the edge e assigned to the current thread p_id;

[0081] (4). For edge e(i,j), obtain the partition c corresponding to points i and j according to the new partition data P_new. i ,c j ;

[0082] (5). The element m[p_id](c) of the current thread's submatrix i ,c j ), m[p_id](c j ,c i Add 1 to each;

[0083] (6). Add the current edge weight |e| to the element of the weight array corresponding to the thread;

[0084] (7). Return to step (3) until all edges have been processed;

[0085] (8). For all non-zero elements m[p_id](i,j) of the obtained submatrices, add m[p_id](i,j) to the corresponding modularity matrix E(i,j);

[0086] (9). Summing w[] gives the total weight W = 44;

[0087] (10). E multiplied by

[0088] (11). Return the modularity matrix.

[0089] Step 3: Calculate the modularity M using graph algebra based on the obtained modularity matrix. The specific steps are as follows:

[0090] (1). Calculate the trace Tr(E) of the modularity matrix E = 0.840909;

[0091] (2). Calculate the modulus ||E|| of the squared modularity matrix E. 2 ||=0.492510;

[0092] (3). The modularity M is calculated as Tr(E) - ||E 2 ||=0.348399;

[0093] (4). The module degree returned is M = 0.348399.

[0094] The above are two preferred embodiments of the present invention, and those skilled in the art can implement or use the present invention in this manner. Meanwhile, those skilled in the art can modify the technical solutions or embodiments provided by the present invention. Therefore, the present invention is not limited to the embodiments described herein, and simple modifications to the embodiments without departing from the technical solutions of the present invention are all within the protection scope of the present invention.

Claims

1. A community detection evaluation method based on graph algebra, characterized in that: For a given graph G and partition data P, the method comprises: Step one, remapping the partition data P to obtain new partition data P_new, wherein the partition id of the new partition data P_new is continuous; Step two, constructing a modularity matrix E in parallel without locking according to the edge set data of the graph G and the new partition data P_new obtained in step one; Step three, performing corresponding graph calculation on the modularity matrix E obtained in step two to obtain a modularity M, which is a community detection evaluation result based on graph algebra; In step one, the data structure of the partition data P is a one-dimensional array, the subscript of the one-dimensional array is the node vrt_id in the graph G, and the value at the subscript of the node vrt_id in the graph G is the partition prt_id corresponding to the current node. The partition data P is essentially a mapping from vrt_id to prt_id; In step one, the method for remapping the partition data P to obtain new partition data P_new is: (1) Construct a mapping between the old partition id and the new partition id, and assign the maximum partition max_id as 0; (2) Start accessing the prt_id in the old partition data; If the current prt_id has not been remapped, the current prt_id is mapped to max_id, and max_id is incremented by 1; If the current prt_id has been remapped, the current prt_id is not remapped; (3) Return to step (2) until all nodes are traversed to obtain new partition data P_new; In step two, the data structure of the graph G is an edge set array, a two-dimensional array, or a structure that can loop through the edges; In step two, the method for constructing the modularity matrix E is: (1) Initialize the sub-matrix, the number of which is the maximum thread number; (2) Distribute all edges in the graph G, and different threads process the distributed edges in parallel; (3) Traverse the edges distributed to the current thread; (4) For edge (i, j), according to the new partition data, the partition ids corresponding to i and j are c(i), c(j), the corresponding element e(c(i), c(j)) of the sub-matrix of the current thread, and e(c(i), c(j)) is added to the edge weight value. For an unweighted graph, add 1, and i and j are the two node ids of the edge; (5) Return to step (3) until all edges are processed; (6) Traverse all non-zero elements e(i, j) obtained to add e(i, j) to the corresponding modularity matrix element E(i, j); (7) Divide all non-zero elements of the matrix by 2 times the total weight of the edges to obtain the modularity matrix E; The corresponding graph calculation of the modularity matrix E in step three refers to first calculating the trace of the modularity matrix E , then calculating the modulus of the square of the modularity matrix E , and finally obtaining the modularity M= .

Images