Local community detection method and system based on noise graph stream, terminal and medium

Abstract

The application discloses a local community detection method based on a noise graph stream, a system thereof, a terminal and a medium. The detection method is used for reading edges in each graph in a graph set in a data stream mode from a file system, a collection of the edges is defined as a graph stream, and a community to be detected is obtained by detecting each edge in the graph stream. The detection method comprises the following steps: defining a collection of edges and a collection of nodes; detecting noise edges; expanding a collection of nodes of the community to be detected; and pruning the expanded collection of nodes to form the community to be detected. The application greatly reduces memory overhead by means of a streaming reading mode, makes the method capable of being applied to a large-scale data graph application scenario, and reduces the influence of noise edges on the accuracy of community detection by detecting the noise edges and defining a robust node membership index, so that higher accuracy is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of complex network community detection, and particularly relates to a local community detection method and system based on noise graph stream, a terminal and a medium. BACKGROUND

[0002] In reality, complex networks can be found everywhere, such as social networks, the World Wide Web, power grids, biological protein networks, etc. These complex networks are not composed of randomly connected vertices (i.e. nodes), and vertices of the same type are often connected together to form a compact community. The community structure usually reflects the specific patterns or functions existing in the network. For example, in biological networks, biological communities represent modules of species with common habits; in WWW (World Wide Web) networks, it represents web pages or attributes with the same or similar topic categories, and communities in social networks represent social groups with the same interests, regions, backgrounds. Community discovery has extremely important theoretical value and practical significance in detecting the functional characteristics, internal topology structure of complex networks, and predicting network behavior. It can effectively guide researchers to conduct more in-depth exploration and research on the organization mode and operation mechanism of complex systems.

[0003] However, on the one hand, the explosive growth of the scale of current complex networks leads to the problem of huge memory overhead of traditional community detection algorithms that load all graphs into memory. Even for local community detection, it usually takes a long time to get the results.

[0004] On the other hand, existing graph stream community detection methods assume that stream data is complete and clean, which rarely occurs in the real world. Social spam in social networks or abnormal transaction behavior in banking and financial networks can introduce noise links, resulting in graph stream data obtained containing a large number of noisy edges. These noisy edges show incorrect link structure information and are easy to mistakenly add nodes outside the community to the community when expanding the community, thus causing community pollution. SUMMARY

[0005] In order to solve the problem of huge memory overhead of traditional community detection algorithms and the noise problem, the present application provides a local community detection method and system based on noise graph stream, a terminal and a medium.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A local community detection method based on noise graph stream, which is used to read edges in each graph in a graph set from a file system, the set of edges is defined as a graph stream, and a community to be detected is obtained by detecting each edge in the graph stream, the detection method comprising the following steps:

[0008] S1, define the edge set S and the node set K

[0009] read the graph stream from the file system in a data stream manner, the graph stream is the set S: S = {e1, e2, …, en}, m represents the serial number of the edge e, each edge e has two nodes; m

[0010] specify a fixed number of nodes as seed nodes, thereby forming a plurality of initial nodes for detecting community expansion: k1, k2, …, kn, n represents the serial number of the seed node k, and the node set K is constituted: K = {k1, k2, …, kn}; n n

[0011] S2, detect noise edges

[0012] For each edge read in, calculate the similarity between the two nodes in each edge and the distance of each edge to the community to be detected;

[0013] Compare the similarity and the distance with the pre-set similarity threshold range and distance threshold range respectively, if the similarity is not within the similarity threshold range, it is determined that the corresponding edge is irrelevant to the community to be detected, and is removed from the set S, otherwise it is determined that the corresponding edge affects the detection result of the community to be detected, and is retained in the set S; if the distance is not within the distance threshold range, it is determined that the corresponding edge is irrelevant to the community to be detected, and is also removed from the set S, otherwise it is also determined that the corresponding edge affects the detection result of the community to be detected, and is also retained in the set S;

[0014] S3, expand the node set K of the community to be detected

[0015] For each edge retained in the set S through noise edge detection, it is also judged synchronously: if one node of the edge is already contained in the node set K, the other node of the corresponding edge is added to the node set K and a membership degree is calculated for the newly added node;

[0016] Return to step S2 until all edges in the set S are detected for noise edges and the corresponding nodes are expanded, and the membership degree calculation for the corresponding newly added nodes is completed;

[0017] S4, prune the expanded node set K

[0018] Sort the membership degrees of all nodes in the node set K from large to small, and only retain the corresponding nodes in the node set K whose membership degrees are within the membership degree threshold range, and the final node set K is the community to be detected.

[0019] ​​​As a further improvement of the above scheme, the method for calculating the similarity comprises the following steps:

[0020]

[0021] wherein u and v represent two nodes in an edge respectively;

[0022] N(u) and N(v) represent the neighbor nodes of the two nodes u and v respectively;

[0023] N(u)∩N(v)| represents the number of the same neighbor nodes of the two nodes u and v;

[0024] N(u)∪N(v)| represents the number of all the neighbor nodes of the two nodes u and v.

[0025] As a further improvement of the above scheme, the method for calculating the distance comprises the following steps:

[0026] First, define dist(u,k) as the distance between node u and seed node k:

[0027]

[0028] wherein dist(v,k) represents the distance between node v and seed node k;

[0029] N(u) represents the neighbor nodes of node u;

[0030] If node u is a seed node, the distance between them is considered as 1; if there is no path between node u and seed node k, indicating that node u cannot reach seed node k, the distance between them is considered as ∞; otherwise, the distance between node u and seed node k is considered as the distance between the neighbor nodes of node u and seed node k + 1;

[0031] Then, define the detected community C, i.e. the synchronously updated node set K, as the expansion of the node set K in the last step; define the distance between one node in an edge and the community as: dist(u,C) = Min{dist(u,k)|k∈K}; the operation Min{dist(u,k)|k∈K} represents that for node u, the distances between it and the seed nodes k in the node set K in the last step are calculated respectively, and the smallest distance value is returned as the distance between one node and the community;

[0032] Finally, the distance of an edge to a detected community is defined as dist (e, C) = Max {dist (u, C), dist (v, C)}, where dist (u, C) and dist (v, C) represent the distance of the nodes u and v of the edge to the detected community C, and the Max {dist (u, C), dist (v, C)} operation represents returning the maximum value of the distance of the nodes u and v to the detected community C as the distance of the edge to the community.

[0033] As a further improvement of the above-mentioned solution, the calculation method of the membership degree comprises the following steps:

[0034] First, the similarity Sim (u, C) of the node u to the detected community C, i.e., the set of nodes K updated synchronously, is calculated, which is represented as the sum of the similarity of the node u to each node k in the community C divided by the size of the community C, and the formula is defined as:

[0035]

[0036] In the formula, ∑ v∈C Sim (u, v) represents the sum of the similarity of the node u to each node v in the community C, and |C| represents the size of the detected community;

[0037] Then, the similarity of the neighbor nodes of the node u to the community C is defined as:

[0038]

[0039] In the formula, K represents the set of nodes K in the previous step, Sim (v, C) represents the similarity of the neighbor nodes of the node u to the detected community C, and ∑ v∈N(u) Sim (v, C) represents the sum of the similarity of all the neighbor nodes of the node u to the detected community C, and |N (u)| represents the number of all the neighbor nodes of the node u;

[0040] Finally, the membership degree of the node u to the detected community C is defined as:

[0041]

[0042] In the formula, Sim (u, C) represents the similarity of the node u to the community C, Sim (v, C) represents the similarity of the neighbor nodes of the node u to the community.

[0043] The application also provides a local community detection system based on a noise graph stream, which is used for reading edges in each graph in a graph set from a file system, the set of the edges is defined as a graph stream, a detected community is obtained by detecting each edge in the graph stream, and the detection system comprises:

[0044] ​The initialization module defines the set of edges S and the set of nodes K: it reads the graph stream from the file system as a data stream, where the graph stream is the set S: S = {e1, e2, ..., e...} m}, where m represents the index of edge e, and each edge e has two nodes; a fixed number of nodes are specified as seed nodes, thus forming multiple initial nodes for detecting community expansion: k1, k2, ..., k n Let n represent the index of the seed node k, forming a node set K: K = {k1, k2, ..., k} n};

[0045] The noise edge detection module is used to detect noise edges: For each edge read in, it calculates the similarity between the two nodes in each edge and the distance from each edge to the community to be detected; it compares the similarity and the distance with preset similarity threshold ranges and distance threshold ranges respectively. If the similarity is not within the similarity threshold range, the corresponding edge is determined to be unrelated to the community to be detected and is removed from set S; otherwise, the corresponding edge is determined to affect the detection result of the community to be detected and is retained in set S. Similarly, if the distance is not within the distance threshold range, the corresponding edge is determined to be unrelated to the community to be detected and is also removed from set S; otherwise, the corresponding edge is also determined to affect the detection result of the community to be detected and is retained in set S.

[0046] The node expansion module is used to expand the node set K of the community to be detected: For each edge in set S that is retained through noise edge detection, it also simultaneously judges: if one node of the edge is already included in node set K, then the other node of the corresponding edge is added to node set K and a membership degree is calculated for the newly added node; the noise edge detection module is called until noise edge detection and corresponding node expansion are performed on all edges in set S, and the membership degree of the newly added node is calculated.

[0047] The pruning module is used to prune the expanded node set K: sort all nodes in node set K from largest to smallest membership degree, and retain only the corresponding nodes whose membership degree is within the membership degree threshold range in node set K. The final node set K is the community to be detected.

[0048] As a further improvement to the above scheme, the noise edge detection module includes a similarity calculation submodule; the similarity calculation submodule calculates the similarity using the similarity score calculation formula Sim(u,v):

[0049]

[0050] In the formula, u and v represent two nodes in an edge, respectively;

[0051] N(u) and N(v) represent the neighbor nodes of nodes u and v respectively;

[0052] |N(u)∩N(v)| represents the number of the same neighbor nodes of nodes u and v;

[0053] |N(u)∪N(v)| represents the number of all the neighbor nodes of nodes u and v.

[0054] As a further improvement of the above scheme, the noise edge detection module comprises a distance calculation submodule, and the distance calculation submodule calculates the distance as follows:

[0055] First, define dist(u,k) as the distance between node u and seed node k:

[0056]

[0057] In the formula, dist(v,k) represents the distance between node v and seed node k;

[0058] N(u) represents the neighbor nodes of node u;

[0059] If node u is a seed node, it is considered that the distance between them is 1; if there is no path between node u and seed node k, indicating that node u cannot reach seed node k, it is considered that the distance between them is ∞; otherwise, it is considered that the distance between node u and seed node k is the distance of the neighbor node of node u to seed node k + 1;

[0060] Then, define the detected community C, i.e. the synchronously updated node set K, as an extension of the node set K of the previous step; define the distance from one of the nodes in the edge to the community as: dist(u,C)=Min{dist(u,k)|k∈K}; the operation Min{dist(u,k)|k∈K} means that for node u, the distances between it and the seed nodes k in the node set K of the previous step are calculated respectively, and the minimum distance value is returned as the distance from one node to the community;

[0061] Finally, define the distance from the edge to the detected community as dist(e,C)=Max{dist(u,C),dist(v,C)}, where dist(u,C) and dist(v,C) represent the distances from the nodes u and v of the edge to the detected community C respectively, and the operation Max{dist(u,C),dist(v,C)} means that the maximum value of the distances from the nodes u and v to the detected community C is returned as the distance from the edge to the community.

[0062] As a further improvement of the above scheme, the node expansion module further comprises a membership calculation submodule, and the membership calculation submodule calculates the membership as follows:

[0063] First, the similarity Sim(u, C) between node u and the detected community C, i.e., the set of nodes K updated synchronously, is calculated, which is represented as the sum of the similarity between node u and each node k in community C divided by the size of community C, and the formula is defined as:

[0064]

[0065] In the formula, ∑ v∈C Sim(u, v) represents the sum of the similarity between node u and each node v in community C, and |C| represents the size of the detected community;

[0066] Then, the similarity Sim(v, C) between the neighbor nodes of node u and the detected community C is calculated, and the formula is defined as:

[0067]

[0068] In the formula, K represents the set of nodes K in the previous step, Sim(v, C) represents the similarity between the neighbor nodes of node u and the detected community C, and ∑ v∈N(u) Sim(v, C) represents the sum of the similarity between all neighbor nodes of node u and the detected community C, and |N(u)| represents the number of all neighbor nodes of node u.

[0069] Finally, the membership of node u to the detected community C is defined as:

[0070]

[0071] In the formula, Sim(u, C) represents the similarity between node u and community C, Sim(v, C) represents the similarity between the neighbor nodes of node u and community C.

[0072] The application also provides a computer terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of any of the above local community detection methods based on a noise graph stream.

[0073] The application also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of any of the above local community detection methods based on a noise graph stream.

[0074] ​Compared with existing technologies, this invention takes into account the huge memory overhead faced by traditional local community detection methods when dealing with large-scale datasets, and the fact that the detection accuracy is easily affected by noisy edges in the graph. It proposes a method for local community detection in graphs with noisy edges. This method reduces the impact of noisy edges on detection accuracy by detecting noisy edges and defining a robust membership index. Furthermore, it greatly reduces memory overhead by using streaming readout, making the method applicable to large-scale data graph application scenarios. Attached Figure Description

[0075] Figure 1 This is a flowchart of a local community detection method based on noise graph flow provided in Embodiment 1 of the present invention.

[0076] Figure 2 To and Figure 1 A schematic diagram of the module structure of a local community detection system based on noise graph flow, corresponding to the local community detection method based on noise graph flow. Detailed Implementation

[0077] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0078] Example 1

[0079] Please see Figure 1 This is a flowchart of a local community detection method based on noisy graph flow provided in Embodiment 1 of the present invention. The local community detection method based on noisy graph flow of the present invention reads edges from each graph in a graph atlas from a file system. The set of these edges is defined as a graph flow, and the community to be detected is obtained by detecting each edge in the graph flow. In application, the local community detection method can be configured as a software system, such as a local community detection system based on noisy graph flow that corresponds one-to-one with the steps of the local community detection method based on noisy graph flow. Please refer to... Figure 2 , its with Figure 1 The diagram shows the module structure of a local community detection system based on noisy graph flow, corresponding to the local community detection method in the Chinese diagram. The local community detection system includes: initialization module 1, noise edge detection module 2, node expansion module 3, and pruning module 4.

[0080] The local community detection method of this invention mainly includes four steps: S1, defining the set of edges S and the set of nodes K; S2, detecting noisy edges; S3, expanding the set of nodes K of the community to be detected; S4, pruning the expanded set of nodes K. Each step will be described in detail below.

[0081] S1, define the edge set S and the node set K

[0082] This step can be performed by the initialization module 1 of the local community detection system. The method of defining the set S and the node set K includes the following steps:

[0083] S11, read the graph stream in the form of data stream from the file system, the graph stream is the set S: S = {e1, e2, …, en}, m represents the serial number of the edge e, each edge e has two nodes; m

[0084] S12, specify a fixed number of nodes as seed nodes, thereby forming a plurality of initial nodes for detecting community expansion: k1, k2, …, kn, n represents the serial number of the seed node k, and the node set K is formed: K = {k1, k2, …, kn}. n n

[0085] The node can also be referred to as a vertex. The purpose of the present application is to obtain a brand new node set K, so the node set K is a variable, not a constant, and the final node set K is the purpose of the present application: the community to be detected. In order to obtain the community to be detected that we need, the main purpose of step S1 is to set the starting point. At this time, the nodes contained in the node set K are nodes that are recognized as truly belonging to the community to be detected after artificial analysis, so they are defined as seed nodes. The nodes that are continuously added to the node set K in the later data processing process are also assigned as seed nodes, but there is a difference because they may not be nodes that the community to be detected truly needs. The initial nodes set by initialization are nodes that the community to be detected truly needs.

[0086] In this embodiment, a fixed number of seed nodes are specified as initial nodes for community expansion. Suppose that the initial seed nodes are three, and the set formed thereby is K = {k1, k2, k3}. K represents the initial seed node set, and k1, k2, and k3 represent the initial three seed nodes. The graph containing noise edges read from the file system in the form of data stream can be abstracted as S = {e1, e2, e3, e4, …}. S represents the set of edges in the read graph, and ei represents the i-th edge read. i

[0087] ​​​​The application converts a graph into a graph stream composed of edges in the graph, reads and processes only one edge in the graph stream each time, expands a community by continuously processing edge structure information in the graph stream, and obtains a final detection result, i.e., a community to be detected. This graph stream type processing of a large-scale graph can greatly save memory overhead, and enable a local community detection algorithm to process a large-scale network. Therefore, the application solves the problem of huge memory overhead faced by a traditional local community detection algorithm when facing a large-scale graph by reading a graph stream.

[0088] S2, detecting a noise edge

[0089] This step can be performed by a noise edge detection module 2 of the local community detection system. The detection method of the noise edge comprises the following steps.

[0090] S21, for each read edge, calculating a similarity between two nodes in each edge and a distance of each edge to a community to be detected;

[0091] S22, comparing the similarity and the distance with a pre-set similarity threshold range and distance threshold range respectively, if the similarity is not in the similarity threshold range, determining that the corresponding edge is irrelevant to the community to be detected, and removing the edge from the set S, otherwise, determining that the corresponding edge affects the detection result of the community to be detected, and retaining the edge in the set S; if the distance is not in the distance threshold range, determining that the corresponding edge is irrelevant to the community to be detected, and removing the edge from the set S, otherwise, determining that the corresponding edge affects the detection result of the community to be detected, and retaining the edge in the set S.

[0092] The calculation method of the similarity in step S21 is that the similarity is calculated by using a similarity score calculation formula Sim(u, v).

[0093]

[0094] In the formula, u and v respectively represent two nodes in an edge;

[0095] N(u) and N(v) respectively represent neighbor nodes of two nodes u and v;

[0096] N(u)∩N(v)| represents the number of same neighbor nodes of two nodes u and v;

[0097] N(u)∪N(v)| represents the number of all neighbor nodes of two nodes u and v.

[0098] In this embodiment, for an edge, we can further represent e=(u, v), where u and v respectively represent two vertices in an edge. We denote a similarity score between vertex u and vertex v as Sim(u, v). where N(u) and N(v) represent the neighbor nodes of vertex u and vertex v respectively, |N(u)∩N(v)| represents the number of the same neighbor nodes of vertex u and vertex v, and |N(u)∪N(v)| represents the number of all the neighbor nodes of vertex u and vertex v. The greater the similarity score between vertex u and vertex v is, the higher the possibility of the existence of the edge between the two vertices is, i.e. the smaller the possibility of the edge being a noise edge is, and vice versa.

[0099] The method for calculating the distance in step S21 comprises the following steps:

[0100] First, define dist(u, k) as the distance from node u to seed node k:

[0101]

[0102] where dist(v, k) represents the distance from node v to seed node k;

[0103] N(u) represents the neighbor nodes of node u;

[0104] If node u is a seed node, the distance between them is considered to be 1; if there is no path between node u and seed node k, indicating that node u cannot reach seed node k, the distance between them is considered to be ∞; otherwise, the distance from node u to seed node k is considered to be the distance from the neighbor node of node u to seed node k + 1;

[0105] Then, define the detected community C, i.e. the synchronously updated node set K, as being extended from the node set K in the previous step; define the distance from one of the nodes in the edge to the community as dist(u, C) = Min{dist(u, k) | k∈K}; the operation Min{dist(u, k) | k∈K} means that for node u, the distances from node u to the seed nodes k in the node set K in the previous step are calculated respectively, and the smallest distance value is returned as the distance from one node to the community;

[0106] Finally, define the distance from the edge to the detected community as dist(e, C) = Max{dist(u, C), dist(v, C)}, where dist(u, C) and dist(v, C) represent the distances from the nodes u and v of the edge to the detected community C respectively, and the operation Max{dist(u, C), dist(v, C)} means that the maximum value of the distances from nodes u and v to the detected community C is returned as the distance from the edge to the community.

[0107] The pre-set similarity threshold range and distance threshold range can not be a range, but a single value, such as the similarity threshold and the distance threshold in the embodiment. The pre-set similarity threshold and distance threshold are as follows: we pre-set a suitable vertex similarity threshold γ sim and a distance threshold λ dist , the vertex similarity threshold γ sim is used to determine whether the edge is a noise edge, if the similarity Sim(u, v) of the two vertices of the edge is greater than the similarity threshold γ sim , it is indicated that the edge is not a noise edge, otherwise it is considered to be a noise edge, and then discarded for the next step. At the same time, we should also consider the distance of the edge to the detected community C, if the distance is farther, the possibility of the edge being a community edge is lower, and removing such a long-distance edge is beneficial to improve the accuracy of the community detection method and the execution efficiency of the detection method. If the distance dist(e, C) of an edge e to the currently detected community C is greater than the distance threshold λ dist , it is indicated that the edge is far away from the detected community C and should be removed for processing, otherwise it is considered to be a community edge for community expansion.

[0108] The noise edges and edges irrelevant to the detected community are removed through the comparison results as follows: first, for the input edge e=(u, v), we calculate the similarity Sim(u, v) between the vertices, which represents the probability of forming an edge between the two vertices, and the distance dist(e, C) of the edge to the detected community C, then the edges not meeting the screening condition are removed, and the edges meeting the condition are further processed. The screening condition is as follows: if Sim(u, v)≥γ sim and dist(e, C)≤λ dist , it is indicated that the edge is a non-noise edge related to the detected community, and the edge is used for community expansion, otherwise the edge is discarded.

[0109] The technical scheme of the present application solves the huge memory overhead problem of the traditional local community detection algorithm when facing large-scale graphs by reading the graph stream, and on this basis, considering that the graph data obtained in real life, whether crawled by a web crawler or collected manually, usually contains a large number of noise edges, the present application belongs to a community detection method and system under a graph stream containing noise edges, which makes the method and system better applied to real life, solving the huge noise problem faced by the traditional community detection algorithm. Therefore, the present application can solve the huge memory overhead problem and the noise problem faced by the traditional community detection algorithm.

[0110] S3, expanding the node set K of the detected community

[0111] This step can be performed by the node expansion module 3 of the local community detection system. The method for expanding the nodes of the community to be detected includes the following steps:

[0112] S31. For each edge retained in set S through noise edge detection, a judgment is also made simultaneously: if one node of the edge is already included in the node set K, then the other node of the corresponding edge is added to the node set K and a membership degree is calculated for the newly added node.

[0113] S32. Return to step S2 until noise edge detection and corresponding node expansion are performed on all edges in set S, and the membership degree of the newly added nodes is calculated.

[0114] This step has a unique characteristic: during noise edge detection, the corresponding steps are completed sequentially while each edge is being detected. It's not that the expansion of nodes in the target community is completed all at once after noise edge detection, nor is the membership calculation done all at once after noise edge detection. Instead, while performing noise edge detection on an edge, as soon as it's determined that the edge should be retained in set S, the expansion of nodes in the target community can be performed simultaneously. Only after node expansion and membership calculation are complete does the noise edge detection for the next edge begin. This cycle continues until noise edge detection and corresponding node expansion have been performed on all edges in set S, and the membership calculation for the newly added nodes is completed. Only then does the entire program loop exit and proceed to the next step, S4.

[0115] Therefore, steps S2 and S3 are synchronized and parallel, which is crucial as it determines whether traditional community detection algorithms can effectively address their significant memory overhead and noise issues. This invention aims for speed, low memory consumption, and low noise. Throughout the process, all edges in set S (i.e., all edges in the graph flow) are detected only once, resulting in fast processing speed. This approach also eliminates the need for caching, thus minimizing memory usage. Some may question the accuracy, arguing that "useful" edges are discarded. However, this invention focuses on studying web pages or attributes with the same or similar themes, and social groups with shared interests, regions, and backgrounds. The nodes in these groups are closely related, and the nodes removed by the method of this invention are necessarily not closely related. Therefore, under these specific research conditions, the processing accuracy of this invention is actually greatly improved.

[0116] In this embodiment, if one vertex of an edge is already included in the community, the other node is added to the community as follows: For an edge e = (u, v), if one of the nodes u already exists in the detected community C, we consider that the node v connected to u should also be added to the community C.

[0117] The calculation method of the membership in step S31 includes the following steps:

[0118] First, the similarity Sim(u, C) of node u and the detected community C, i.e., the set of nodes K updated synchronously, is calculated, which is represented as the sum of the similarity of node u and each node k in the community C divided by the size of the community C. The formula is defined as:

[0119]

[0120] In the formula, ∑ v∈C Sim(u, v) represents the sum of the similarity of node u and each node v in the community C, and |C| represents the size of the detected community;

[0121] Then, the similarity of the neighbor nodes of node u and the community C is defined as:

[0122]

[0123] In the formula, K represents the node set K of the previous step, Sim(v, C) represents the similarity of the neighbor nodes of node u and the detected community C, and ∑ v∈N(u) Sim(v, C) represents the sum of the similarity of all neighbor nodes of node u and the detected community C, and |N(u)| represents the number of all neighbor nodes of node u;

[0124] Finally, the membership of node u to the detected community C is defined as:

[0125]

[0126] In the formula, Sim(u, C) represents the similarity of node u and the community C, Sim(v, C) represents the similarity of the neighbor nodes of node u and the community.

[0127] In this embodiment, in order to distinguish the membership strength between the added node and the community, a membership index is calculated for each node added to the community, which is specifically: for the detected community C, if node v in edge e=(u, v) is already in the detected community C, then node u should be added to the detected community, and then the membership of node u to the detected community C is calculated. The process of calculating the membership is as follows: first, the similarity Sim(u, C) of node u and the detected community C is calculated, which is represented as the sum of the similarity of node u and each node k in the community C divided by the size of the community C. The formula is defined as: In the formula, ∑ v∈C ​Sim(u, v) represents the sum of the similarity of node u with each node k in the community C, and |C| represents the size of the detected community. Then, the membership of a node to a community is also related to the similarity of its neighbor nodes to the community, that is, if the neighbor nodes of a node have a higher similarity to the community C, the node is more likely to be a node in the community C. We define the membership of a node to a community as follows: Sim(u, v) represents the sum of the similarity of node u with each node k in the community C, and |C| represents the size of the detected community. Then, the membership of a node to a community is also related to the similarity of its neighbor nodes to the community, that is, if the neighbor nodes of a node have a higher similarity to the community C, the node is more likely to be a node in the community C. We define the membership of a node to a community as follows: where K represents the seed node set, Sim(v, C) represents the similarity of the neighbor nodes of node u to the detected community C, and ∑ v∈N(u) Sim(v, C) represents the sum of the similarity of all neighbor nodes of node u to the detected community C, and |N(u)| represents the number of all neighbor nodes of node u. Specifically, if node u is a seed node, it must belong to the community and its similarity is 1, if it is not a seed node, the similarity of the neighbor nodes of node u to the detected community C is the sum of the similarity of all neighbor nodes of node u to the detected community C divided by the number of all neighbor nodes of node u. Finally, we define the membership of node u to the detected community C as follows: where Sim(u, C) represents the similarity of node u to the community C, Sim(u, v) represents the similarity of the neighbor nodes of node u to the community. We take their product as a measure of the membership of node u to the detected community C. The larger the membership index of a node indicates that the node is more likely to belong to this community, and vice versa.

[0128] S4, pruning the expanded node set K

[0129] This step can be performed by the pruning module 4 of the local community detection system, and the pruning method includes the following steps:

[0130] The membership of all nodes in the node set K is sorted from large to small, and only the corresponding nodes with a membership within the membership threshold range are retained in the node set K. The final node set K is the detected community.

[0131] In this embodiment, the pruning of the community is as follows: after all the noisy edges are read, the detected community is expanded from the seed node set K to the node set C. For each node in C, there is a membership index. We sort all the nodes in descending order of membership, and remove the nodes with small membership close to 0 from the detected node set C. In this method, we retain all the nodes with a membership greater than 0 as the result output.

[0132] The method has the advantages that the technical scheme of the application considers the huge memory cost of the traditional local community detection method when facing large-scale data sets, and the detection accuracy is easily affected by the noise edges in the graph, and the method for local community detection under the graph stream containing noise edges is invented, the method reduces the influence of noise edges on the detection accuracy by detecting the noise edges and defining a robust membership index, and the memory cost is greatly reduced by the streaming reading method, so that the method can be applied to large-scale data graph application scenarios.

[0133] Embodiment 2

[0134] Based on the embodiment 1, the embodiment further provides a computer terminal, which is an entity computing device for implementing the scheme in the embodiment 1. The computer terminal comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the local community detection method based on the noise graph stream in the embodiment 1 are implemented, and then the community to be detected is output.

[0135] The computer device can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including a single server or a server cluster composed of multiple servers), etc. which can execute programs. The computer device of the embodiment at least includes but is not limited to a memory and a processor which can be connected to each other by a system bus.

[0136] In this embodiment, the memory (i.e., the computer readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the memory can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device, etc. In addition, the memory can also be used to temporarily store various data that have been output or will be output. Regardless, the computer readable storage medium stores a computer program that, when executed by the processor, implements the steps of the local community detection method based on a noise graph stream as in Embodiment 1, and then outputs the community to be detected.

[0137] The processor can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of the computer device.

[0138] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, and improvement within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for detecting local communities based on noise graph stream, which is used to read edges in each graph in a graph set from a file system, the set of edges is defined as a graph stream, and a community to be detected is obtained by detecting each edge in the graph stream, characterized in that, The file system is a file system in a complex network; the complex network includes a social network, a world wide web, a power network, and a biological protein network; and the detection method comprises the following steps: S1, defining a set S of edges and a set K of nodes The graph stream is read from the file system as a data stream, and the graph stream is a set S: S = {e1, e2, ..., e...} m }, where m represents the index of edge e, and each edge e has two nodes; A fixed number of nodes are designated as seed nodes, thereby forming a plurality of initial nodes for detecting community expansion: k1, k2, …, k n n represents the serial number of the seed node k, and the node set K is formed: K = {k1, k2, …, k n} S2, detecting noise edges For each read-in edge, the similarity between the two nodes in each edge and the distance of each edge to the community to be detected are calculated; The similarity and the distance are compared with a pre-set similarity threshold range and a distance threshold range, respectively; if the similarity is not within the similarity threshold range, it is determined that the corresponding edge is irrelevant to the community to be detected, and the corresponding edge is removed from the set S; otherwise, it is determined that the corresponding edge affects the detection result of the community to be detected, and the corresponding edge is retained in the set S; if the distance is not within the distance threshold range, it is determined that the corresponding edge is irrelevant to the community to be detected, and the corresponding edge is also removed from the set S; otherwise, it is also determined that the corresponding edge affects the detection result of the community to be detected, and the corresponding edge is also retained in the set S; S3, expanding the set K of nodes of the community to be detected For each edge retained in the set S through noise edge detection, it is also judged synchronously: if one node of the edge is already contained in the set K of nodes, the other node of the corresponding edge is added to the set K of nodes, and a membership degree is calculated for the newly added node; Step S2 is returned until all edges in the set S are detected by noise edge detection and the corresponding nodes are expanded, and the membership degree calculation of the corresponding newly added nodes is completed; S4, pruning the expanded set K of nodes The membership degrees of all nodes in the set K of nodes are sorted from large to small, and only the corresponding nodes with the membership degrees within a membership degree threshold range are retained in the set K of nodes; the final set K of nodes is the community to be detected, and reflects a local community with a specific mode or function in the complex network.

2. The noise graph stream based local community detection method of claim 1, wherein, The similarity is calculated by using a similarity score calculation formula Sim(u, v): In the formula, u and v represent two nodes in an edge; N(u) and N(v) represent the neighbor nodes of the two nodes u and v, respectively; |N(u)∩N(v)| represents the number of the same neighbor nodes of the two nodes u and v; |N(u)∪N(v)| represents the number of all neighbor nodes of the two nodes u and v.

3. The noise graph stream based local community detection method of claim 1, wherein, The distance calculation method comprises the following steps: First, define dist(u, k) to represent the distance from node u to seed node k: In the formula, dist(v, k) represents the distance from node v to seed node k; N(u) represents the neighbor nodes of node u; If node u is a seed node, it is considered that the distance between them is 1; if there is no path between node u and seed node k, it is considered that node u cannot reach seed node k, and the distance between them is considered to be ∞; otherwise, it is considered that the distance from node u to seed node k is the distance from the neighbor node of node u to seed node k + 1. Then, define the detected community C, i.e. the synchronously updated node set K, as an extension of the node set K in the last step; define the distance from one of the nodes in the edge to the community as dist(u, C) = Min{dist(u, k) | k∈K}; the operation Min{dist(u, k) | k∈K} means that for the node u, the distances between the node u and the seed nodes k in the node set K in the last step are calculated respectively, and the minimum distance value is returned as the distance from one node to the community; Finally, define the distance from the edge to the detected community as dist(e, C) = Max{dist(u, C), dist(v, C)}, where dist(u, C) and dist(v, C) represent the distances from the nodes u and v of the edge to the detected community C respectively, and the operation Max{dist(u, C), dist(v, C)} means that the maximum value of the distances from the nodes u and v to the detected community C is returned as the distance from the edge to the community.

4. The noise graph stream-based local community detection method of claim 1, wherein, The calculation method of the membership degree comprises the following steps: First, calculate the similarity Sim(u, C) between the node u and the detected community C, i.e. the synchronously updated node set K, which is represented as the sum of the similarities between the node u and each node k in the community C divided by the size of the community C, and the formula is defined as: In the formula, ∑ v∈C Sim(u, v) represents the sum of the similarity of each node v in the community C with the node u, and |C| represents the size of the detected community. Then, the node u is determined The similarity of the neighbor node of the node u to the community C is represented, and the formula is defined as: In the formula, K represents the node set K of the previous step, Sim(v, C) represents the similarity of the neighbor nodes of node u and the detected community C, ∑ v∈N(u) Sim(v, C) represents the sum of the similarity of all neighbor nodes of node u and the detected community C, and |N(u)| represents the number of all neighbor nodes of node u. Finally, the membership degree of the node u to the detected community C is defined as: In the formula, Sim(u, C) represents the similarity of node u and community C, represents the similarity of the neighbor node of node u and community.

5. A system for local community detection based on noise graph stream, for reading edges in each graph in a set of graphs from a file system, the set of edges defining a graph stream, a community to be detected being obtained by detecting each edge in the graph stream, characterized in that, The file system is a file system in a complex network; the complex network comprises a social network, a World Wide Web, a power network and a biological protein network, and the detection system comprises: An initialization module is used to define a set of edges S and a set of nodes K: a graph stream is read from a file system in a data stream manner, the graph stream is a set S: S = {e1, e2, …, e m}, m represents a serial number of the edge e, each edge e has two nodes; a fixed number of nodes are designated as seed nodes, thereby forming a plurality of initial nodes for detecting community expansion: k1, k2, …, k n , n represents a serial number of the seed node k, and a set of nodes K is constituted: K = {k1, k2, …, k n}. a noise edge detection module for detecting noise edges: for each read-in edge, the similarity between the two nodes in each edge and the distance from each edge to the community to be detected are calculated; the similarity and the distance are compared with a pre-set similarity threshold range and a distance threshold range respectively, if the similarity is not within the similarity threshold range, it is determined that the corresponding edge is irrelevant to the community to be detected, and the corresponding edge is removed from the set S, otherwise it is determined that the corresponding edge affects the detection result of the community to be detected, and the corresponding edge is retained in the set S; if the distance is not within the distance threshold range, it is determined that the corresponding edge is irrelevant to the community to be detected, and the corresponding edge is also removed from the set S, otherwise it is also determined that the corresponding edge affects the detection result of the community to be detected, and the corresponding edge is also retained in the set S; a node expansion module for expanding the node set K of the community to be detected: for each edge retained in the set S through noise edge detection, it is also synchronously determined whether one node of the edge is already contained in the node set K; if yes, the other node of the corresponding edge is added to the node set K, and a membership degree is calculated for the newly added node; the noise edge detection module is called until all edges in the set S are subjected to noise edge detection and corresponding node expansion, and the membership degree calculation of the corresponding newly added node is completed. The pruning module is used for pruning the extended node set K: all the membership degrees of the nodes in the node set K are sorted in descending order, and only the corresponding nodes with the membership degrees within the membership degree threshold range are reserved in the node set K, and the final node set K is the community to be detected, which represents a local community reflecting a specific mode or function in the complex network.

6. The noise map stream-based local community detection system of claim 5, wherein, The noise edge detection module comprises a similarity calculation submodule; the similarity calculation submodule calculates the similarity by using a similarity score calculation formula Sim(u, v): In the formula, u and v represent two nodes in an edge; N(u) and N(v) represent the neighbor nodes of the two nodes u and v, respectively; |N(u)∩N(v)| represents the number of the same neighbor nodes of the two nodes u and v; |N(u)∪N(v)| represents the number of all the neighbor nodes of the two nodes u and v.

7. The noise map stream based local community detection system of claim 5, wherein, The noise edge detection module comprises a distance calculation submodule, and when calculating the distance, the distance calculation submodule: First, define dist(u, k) as the distance from the node u to the seed node k: In the formula, dist(v, k) represents the distance from the node v to the seed node k; N(u) represents the neighbor nodes of the node u; If the node u is a seed node, the distance between them is considered to be 1; if there is no path between the node u and the seed node k, it means that the node u cannot reach the seed node k, and the distance between them is considered to be ∞; otherwise, the distance from the node u to the seed node k is considered to be the distance from the neighbor node of the node u to the seed node k + 1; Then, define the detected community C, that is, the node set K which is synchronously updated, as being extended from the node set K in the last step; define the distance from one node in the edge to the community as dist(u, C) = Min{dist(u, k)|k∈K}; the Min{dist(u, k)|k∈K} operation means that for the node u, the distances from the node u to the seed nodes k in the node set K in the last step are calculated respectively, and the minimum distance value is returned as the distance from one node to the community; Finally, define the distance from the edge to the detected community as dist(e, C) = Max{dist(u, C), dist(v, C)}, where dist(u, C) and dist(v, C) represent the distances from the nodes u and v of the edge to the detected community C respectively, and the Max{dist(u, C), dist(v, C)} operation means that the maximum value of the distances from the nodes u and v to the detected community C is returned as the distance from the edge to the community.

8. The noise map stream based local community detection system of claim 5, wherein, The node expansion module further comprises a membership degree calculation submodule, and when calculating the membership degree, the membership degree calculation submodule: First, calculate the similarity Sim(u, C) between the node u and the detected community C, that is, the synchronously updated node set K, which is represented as the sum of the similarities between the node u and each node k in the community C divided by the size of the community C, and the formula is defined as: where ∑ v∈C Sim(u, v) represents the sum of the similarities between node u and each node k in the community C, and |C| represents the size of the detected community. Then, the node u is determined The similarity of the neighbor node of the node u to the community C is represented, and the formula is defined as: In the formula, K represents the node set K of the previous step, Sim(v, C) represents the similarity of the neighbor nodes of node u and the detected community C, ∑ v∈N(u) Sim(v, C) represents the sum of the similarity of all neighbor nodes of node u and the detected community C, and |N(u)| represents the number of all neighbor nodes of node u. Finally, the membership degree of the node u to the detected community C is defined as: In the formula, Sim(u, C) represents the similarity of node u and community C, represents the similarity of the neighbor nodes of node u and the community.

9. A computer terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the local community detection method based on the noise graph stream according to any one of claims 1 to 4 when executing the program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the steps of the local community detection method based on a noise graph stream according to any one of claims 1 to 4.

Images