A graph theory-based distributed mass node community partitioning method

Abstract

The application discloses a kind of distributed mass node community division methods based on graph theory, belong to social network detection field, by the random sampling of complex network relation, cut is carried out in combination with weighted logarithm GN algorithm, interval calculation module degree and distributed finite iteration, can complete 100,000+ magnitude node relation complex network cutting in short time, and WS small world verification is carried out to the community after cutting, this method can effectively overcome the defects of existing method, can ensure accurate efficient real social network detection, the present application retains logarithm weighted relationship weight and finite random iteration distributed cutting, realize accurate efficient cutting of mass complex network, using small world theory ensures that the clustering coefficient of division community meets real social network.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of social network exploration, and particularly relates to a distributed massive node group division method based on graph theory. BACKGROUND

[0002] In the field of social network exploration, with the development of information technology, people tend to communicate online, and these communication activities often show group characteristics, which constitute the network community. These network communities are hidden, large in scale and complex in structure. In order to solve the problem of network community exploration, people have proposed many algorithms, such as random block model, clustering model, extension model based on graph theory degree centrality, information entropy model, etc. However, these models often have defects in terms of detection accuracy, running efficiency, calculation community convergence and evaluation division accuracy. When facing massive complex network groups, how to quickly and accurately divide the community is the core problem we need to solve.

[0003] In a complex relationship network, the traditional way is to perform full cutting based on iterative external expansion, and it is difficult to evaluate and ensure the cutting effect. SUMMARY

[0004] The technical problem to be solved by the application is the deficiency of the division efficiency and accuracy in the field at present. A distributed massive node group division method based on graph theory is proposed, which retains the distributed cutting of logarithmic weighted relationship weight and limited random iteration, realizes the accurate and efficient cutting of massive complex networks, and finally uses the small world theory to ensure that the clustering coefficient of the divided community meets the real social network.

[0005] To solve the above technical problems, the application adopts the following technical scheme:

[0006] A distributed massive node group division method based on graph theory retains the distributed cutting of logarithmic weighted relationship weight and limited random iteration GN, realizes the accurate and efficient cutting of massive complex networks, and uses the WS small world theory to ensure that the clustering coefficient of the divided community meets the real social network, and specifically includes the following steps:

[0007] Step 1, random sampling: the call communication data is sorted into undirected relationship data with communication frequency;

[0008] Step 2, logarithmic smoothing: when the communication frequency is directly used as the edge weight, the edge weight is logarithmically degraded ln(e i ), in order to avoid the case that the logarithm is 0, all ln(e i )+0.1 are smoothed;

[0009] Step 3, iterative edge removal: calculate the weighted intermediate centrality δ st(e), deleting the edge with the maximum intermediate centrality;

[0010] Weighted intermediate centrality of each edge in the random sample relationship δ st (e), which is specifically calculated as follows:

[0011]

[0012] where σ st (e) represents the number of shortest paths from s to t passing through edge e, σ st represents the number of shortest paths from s to t.

[0013] Step 4, interval sampling modularity: the modularity Q of the entire network is calculated once every M edge deletion, and the current network detection result is retained;

[0014] Step 5, multi-core multi-process distributed calculation: after the iterative calculation of the intermediate centrality, the shortest paths of the network nodes need to be detected again; the shortest paths are encapsulated as a separate method, and all resources of the multi-core server are called for calculation, and all nodes are distributed to different computing sub-machines for distributed calculation; after cutting 2 / 3 edges, the detection result corresponding to the maximum modularity Q is selected;

[0015] Step 6, WS small world verification: each sub-group after detection needs to meet the real-world social structure, and the WS small world clustering coefficient C is used to verify the convergence of the detection result.

[0016] As a further preferred scheme of the distributed massive node group division method based on graph theory, the step 1 is specifically as follows: M edges are randomly taken each time, the intermediate centrality of the edges is calculated, and the modularity is calculated once every M edge deletion, wherein the values of K and M are related to the total number of edges N, and are specifically as follows:

[0017]

[0018] As a further preferred scheme of the distributed massive node group division method based on graph theory, in step 4: the modularity Q is specifically calculated as follows:

[0019]

[0020] where e ij represents the number of edges between community i and community j, a i =∑ i e ij , represents the number of edges connected to community i.

[0021] As a further preferred scheme of the distributed massive node group division method based on graph theory, the step 6 is specifically as follows:

[0022]

[0023] Where N is the number of nodes in the network, k i For node i and other k i Nodes are connected, E i For this k i When the clustering coefficient of a subgroup is greater than 0.4, it is considered to be a valid group that conforms to the real structure. For invalid groups with a clustering coefficient less than 0.4, the group is added to the valid group with the most connections to the group.

[0024] Compared with the prior art, the present invention adopts the above technical solution and has the following technical effects:

[0025] The present invention randomly samples complex network relationships, combines them with the weighted logarithmic GN algorithm for segmentation, calculates modularity at intervals, and performs distributed finite iterations. It can complete the segmentation of complex networks with more than 100,000 node relationships in a short period of time, and performs WS small-world verification on the segmented groups. This method can effectively overcome the shortcomings of existing methods and ensure accurate, efficient and authentic social network detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0027] Figure 1 The diagram shows a complex community with 5560 nodes and 23579 undirected relationships;

[0028] Figure 2 The diagram shows the logarithmic weighting and smoothing of the communication relationship data.

[0029] Figure 3 The figure shows the community subgroups obtained after a finite number of iterative edge removals using random sampling.

[0030] Figure 4 This is a flow chart of a method for dividing distributed massive node groups based on graph theory of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings:

[0032] like Figures 1 to 4The specific embodiments of the present patent will be further described in detail below with reference to the accompanying drawings. From the detailed description of the present patent, those skilled in the art can know the purpose, features and advantages of the present patent.

[0033] Figure 1 The complex community containing 5560 nodes and 23579 undirected relationships is shown, each relationship representing all the communication times between the two nodes within a certain period of time, and the sample edge number K is set to 2358, and the Q interval M is calculated to be 20.

[0034] As Figure 2 shown, we take the logarithm and reduce the weight by order of magnitude, and the distribution is more stable. For the case that the logarithm

[0035] may be 0, we smooth all ln(e i )+0.1. Taking the logarithm ln is a mature operation to reduce the order of magnitude, and adding 0.1 is also a mature smoothing operation, which can be directly understood in data mining.

[0036] Figure 3 The community subgroups obtained after a limited number of iterations of random sampling and edge removal are shown, and the WS small-world clustering coefficient is used for verification and supplement.

[0037] The final module degree with the best cutting effect is obtained. A total of 94 improved sub-communities are obtained, and the verification shows that the communication relationship of each community basically conforms to the real community relationship. The total program running time is about 24 minutes (on two distributed machine resources).

[0038] The specific implementation is as follows: a distributed mass node group division method based on graph theory, as Figure 4 shown, the specific package

[0039] includes the following steps: step 1, random sampling: the call communication data is sorted into undirected relationship data with communication frequency; the specific implementation is as follows: randomly take M edges each time, calculate the intermediate centrality of the edges, and calculate the module degree every M times of edge deletion. This is an optimization and innovation operation, which is equivalent to changing the full operation to a random sampling operation. The principle is to greatly reduce the calculation amount while maintaining the original effect as much as possible. The values of K and M are related to the total number of edges N, and the specific implementation is as follows:

[0040]

[0041] Step 2, logarithmic smoothing: when the communication frequency is directly used as the edge weight, the edge with large frequency definitely has a greater impact, but in the actual relationship, the importance is not much different as long as the frequency exceeds a certain value. In order to optimize the importance of edge weight, the edge weight is reduced by order of magnitude ln(e i), to avoid the case of 0 after taking the logarithm, all ln(e i ) + 0.1 are smoothed;

[0042] Step 3, iterative edge removal: calculate the weighted betweenness centrality δ st (e) of each edge in the random sample relationship, and delete the edge with the maximum betweenness centrality:

[0043]

[0044] Where σ st (e) represents the number of shortest paths from s to t passing through edge e, and σ st represents the number of shortest paths from s to t.

[0045] Step 4, interval sampling modularity; calculate the modularity Q of the entire network every M times of edge deletion, and keep the current network detection result; when there are a large number of edges, small interval calculation of modularity basically does not affect the detection effect, but can greatly reduce the time complexity of calculation:

[0046]

[0047] Where e ij represents the number of edges between community i and community j, and a i = ∑ i e ij represents the number of edges connected to community i.

[0048] Step 5, multi-core multi-process distributed calculation: the most time-consuming part in the entire detection process is the recalculation of the shortest path of the network node after each iteration of betweenness centrality calculation; in order to fully utilize the resources of the cluster computer and improve the calculation efficiency, the shortest path is packaged as a separate method, and all the resources of the multi-core server are called for calculation, and all the nodes are distributed to different computing sub-machines for distributed calculation; single machine operation is converted to distributed operation (equivalent to application innovation), because the shortest path calculation is the most time-consuming part of the entire model, optimizing this part of the calculation can greatly reduce the time complexity of the model calculation. After cutting 2 / 3 edges, the detection result with the maximum modularity Q is selected, which is the best community detection result.

[0049] Step 6, WS small world verification. Each sub-group after detection needs to meet the real-world social structure, in order to verify the convergence of the detection result, the WS small world clustering coefficient C is used for verification:

[0050]

[0051] Where N is the number of nodes of the network. k i is the number of nodes i and another k iA node is connected to E i The number of edges between the k i When the clustering coefficient of a sub-group is greater than 0.4, it is considered to be an effective group conforming to the real structure. For the ineffective group with the clustering coefficient less than 0.4, the group is added to the effective group with which the group is connected most.

[0052] Finally, it should be noted that the above is only the preferred embodiment of the application and is not intended to limit the application. Although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or make equivalent replacements to some technical features. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A graph-based distributed community partitioning method for massive nodes, characterized in that: The distributed cutting of the logarithm weighted relation weight and the limited random iterative GN is reserved to realize the accurate and efficient cutting of the mass complex network, the WS small world theory is used to guarantee that the clustering coefficient of the divided community meets the real social network, and specifically includes the following steps. Step 1, random sampling: the call data connection data is arranged into undirected relation data with call frequency; Step 2, Log smoothing: When the edge weight is directly taken as the edge weight, the edge weight is logarithmically degraded ln(e i ), in order to avoid the case of taking logarithm as 0, all ln(e i )+0.1 are smoothed; Step 3, Iterative edge removal: Calculate the weighted betweenness centrality δ of each edge in the random sample relation st (e) deleting the edge with the largest betweenness centrality; In step 3: Calculate the weighted betweenness centrality δ of each edge in the random sample relation st (e), calculated as follows: where σ st (e) denotes the number of shortest paths from s to t that pass through edge e, σ st denotes the number of shortest paths from s to t. Step 4, interval sampling module degree; the module degree Q of the whole network is calculated once every M times of edge deletion, and the current network detection result is reserved; Step 5, multi-core multi-process distributed calculation: after the iterative calculation of the intermediate centrality, the shortest path of the network node needs to be detected again; the shortest path is encapsulated as a separate method, the resources of the multi-core server are called to calculate, and all nodes are distributed to different computing machines for distributed calculation; after cutting 2 / 3 edges, the detection result corresponding to the maximum module degree Q is selected; Step 6, WS small world verification: each sub-group after detection needs to meet the real world social structure, and the WS small world clustering coefficient C is used to verify the convergence of the detection result.

2. The method of claim 1, wherein: The step 1 is specifically as follows: M edges are randomly taken each time, the intermediate centrality of the edges is calculated, and the module degree is calculated once every M times of edge deletion, wherein the values of K and M are related to the total number of edges N, and are specifically as follows:

3. The method of claim 1, wherein: In step 4, the module degree Q is specifically calculated as follows: where e ij represents the number of edges between community i and community j, a i =∑ i e ij represents the number of edges connected to community i.

4. The graph-based distributed community partitioning method of claim 1, wherein: The step 6 is specifically as follows: Where N is the number of nodes in the network, k i For node i and other k i Nodes are connected, E i For this k i When the clustering coefficient of a subgroup is greater than 0.4, it is considered to be a valid group that conforms to the real structure. For invalid groups with a clustering coefficient less than 0.4, the group is added to the valid group with the most connections to the group.

Images