Community discovery method and device based on structure chart

Abstract

The embodiment of the invention provides a community discovery method based on a structure chart. The method comprises the steps that a community graph is initialized based on a structure graph, and communities are represented by entity vertexes and have community attributes; each community is added with an atomic lock; a plurality of threads are adopted to execute a plurality of rounds of first graph iteration in parallel, and the current round of graph iteration executed by any first thread comprises the steps that for any first entity vertex processed by the first thread, all target communities to which all neighbor entity vertexes meeting a preset relation with the first entity vertex belong are determined, and after atomic locks of all the target communities are obtained, the first entity vertex is processed by the first thread; determining whether the first entity vertex needs to move or not according to the community attribute of each target community, updating the community attribute of the related community when the first entity vertex needs to move, and releasing the atomic lock after updating; and aggregating the entity vertexes in any community into a new entity vertex representing the community according to the community attribution of the entity vertexes after a plurality of rounds of first graph iteration so as to update the community graph.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the computer field, and more particularly, to a community discovery method and device based on a structure graph. Background Art

[0002] In mathematics, a graph is a structure composed of a set of vertices (also called nodes or points) and a set of edges (also called arcs or lines) connecting these vertices, which is used to simulate the relationship between entities. There are two basic elements in the structure graph, namely vertices and edges. Among them, vertices are the basic units in the graph, representing entities or objects. Edges, also called connecting edges, are line segments connecting two vertices in the graph, representing a certain relationship or connection between vertices. Graph data may contain users' private data.

[0003] Structural graphs include homogeneous graphs and heterogeneous graphs. Homogeneous graphs: All vertices in the graph belong to the same category and have the same type of edges. Heterogeneous graphs: The graph contains different types of vertices and edges, and vertices and edges are divided into different categories according to their attributes and meanings.

[0004] Currently, community discovery and other problems are often solved based on graph algorithms. It can be understood that graph algorithms are a series of algorithms executed on graph data. Community discovery is the process of identifying groups of vertices with similarities or dense connections in graph data, where the connection density within these groups is higher than the connection density outside. In the prior art, in community discovery based on structural graphs, due to the complexity of graph algorithm calculations and the huge scale of business graph data, timeliness is often not guaranteed. Summary of the invention

[0005] One or more embodiments of this specification describe a community discovery method and device based on a structure graph, which can greatly improve timeliness.

[0006] In a first aspect, a community discovery method based on a structure graph is provided, wherein the structure graph includes M entity vertices representing M users, and the method includes:

[0007] Based on the structure graph, a community graph is initialized, wherein each community is represented by an entity vertex and has community attributes, wherein the community attributes include a certainty indicator of the corresponding community; and each community is added with an atomic lock;

[0008] A plurality of threads are used to execute several rounds of first graph iterations in parallel, wherein a current round of graph iteration executed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update;

[0009] According to the community affiliations of the entity vertices after the several rounds of first graph iterations, the entity vertices in any community are aggregated into new entity vertices representing the community, so as to update the community graph.

[0010] In a possible implementation, the structural diagram also includes a number of auxiliary vertices representing the optional feature values ​​of each of the N types of features of the user, and there is a connecting edge between any entity vertex and the auxiliary vertex corresponding to the actual feature value of the user, and the deterministic index is determined based on the distribution of feature values ​​of various types of auxiliary vertices associated with each entity vertex in the community.

[0011] In a possible implementation, the entity vertex has a vertex attribute; the vertex attribute is used to indicate the community to which the corresponding entity vertex belongs; and determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs includes:

[0012] Traverse each neighbor entity vertex that satisfies the preset relationship with it, and determine the community where the neighbor entity vertex is located according to the vertex attributes associated with the neighbor entity vertex.

[0013] Furthermore, the method further comprises:

[0014] When movement is required, the vertex attributes of the first entity vertices are updated.

[0015] In a possible implementation manner, determining whether the first entity vertex needs to be moved according to the community attributes of each target community includes:

[0016] According to the community attribute of any target community, determining the gain of the deterministic index after moving the first entity vertex to the target community;

[0017] According to the gains of the target communities, it is determined whether the first entity vertex needs to be moved to one of the target communities.

[0018] In a possible implementation, the method further includes:

[0019] For the updated community graph, several rounds of second graph iterations are performed, wherein at least one round of second graph iterations includes: using multiple threads to execute the several rounds of first graph iterations in parallel.

[0020] In a possible implementation, the initializing the community graph based on the structure graph is performed based on a parallel computing mode.

[0021] In a possible implementation manner, the preset relationship includes: a second-degree neighbor entity vertex having at least one commonly connected auxiliary vertex with the second-degree neighbor entity vertex with the second-degree neighbor entity vertex having multiple commonly connected auxiliary vertices with the second-degree neighbor entity vertex with the second-degree neighbor entity vertex.

[0022] In a possible implementation, the method further includes:

[0023] Based on the community graph, an association graph is initialized; wherein if the first entity vertex and the second entity vertex in the community graph satisfy the preset relationship, the first entity vertex and the second entity vertex are added to the association graph, and a connecting edge between the two is added to the association graph;

[0024] Before determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, the method further includes:

[0025] In the association graph, entity vertices having connecting edges therewith are searched as neighbor entity vertices that satisfy a preset relationship therewith.

[0026] Furthermore, the community graph or the association graph adopts a storage method of compressed row storage (compressed sparse row, CSR) or compressed column storage (compressed sparse column, CSC) to store graph structure information.

[0027] Furthermore, the several rounds of second graph iterations form multi-level communities, and a hierarchical relationship is formed between the community obtained by the N+1th round of second graph iteration and the community obtained by the Nth round of second graph iteration.

[0028] Furthermore, the community attributes also include:

[0029] The set of feature values ​​of various auxiliary vertices associated with each entity vertex in the community;

[0030] The determining a gain of a certainty indicator after moving the first entity vertex to the target community includes:

[0031] Adding the feature values ​​of various auxiliary vertices associated with the first entity vertex to the set of feature values ​​of the target community to obtain an updated set;

[0032] Calculating an updated certainty index according to the update set;

[0033] The gain is obtained by subtracting the certainty index before updating included in the community attribute from the certainty index after updating.

[0034] Furthermore, the community attributes also include:

[0035] The number of entity vertices contained in the community;

[0036] After performing several rounds of second graph iterations, the method further includes:

[0037] Output the community attributes corresponding to each community.

[0038] In a possible implementation, a stopping condition for the first graph iteration is that the communities to which each entity vertex belongs no longer change.

[0039] Furthermore, the stopping condition of the second graph iteration is that the number of communities no longer changes.

[0040] In a second aspect, a community discovery device based on a structure graph is provided, wherein the structure graph includes M entity vertices representing M users, and the device includes:

[0041] An initialization unit, configured to initialize a community graph based on the structure graph, wherein each community is represented by an entity vertex and has community attributes, wherein the community attributes include a certainty indicator of the corresponding community; and each community is added with an atomic lock;

[0042] A first iteration unit is configured to use multiple threads to perform several rounds of first graph iterations in parallel, wherein a current round of graph iterations performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining an atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update;

[0043] The aggregation unit is used to aggregate entity vertices in any community into new entity vertices representing the community according to the community affiliation of the entity vertices after the first iteration unit performs several rounds of first graph iterations, so as to update the community graph.

[0044] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0045] According to a fourth aspect, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0046] Through the method and device provided by the embodiments of this specification, first, based on the structure graph, a community graph is initialized, wherein each community is represented by an entity vertex and has community attributes, and the community attributes include a deterministic indicator of the corresponding community; an atomic lock is added to each community; then multiple threads are used to perform several rounds of first graph iterations in parallel, wherein the current round of graph iteration performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update; finally, according to the community affiliation of the entity vertex after the several rounds of first graph iterations, the entity vertices in any community are aggregated into a new entity vertex representing the community to update the community graph. As can be seen from the above, in the embodiments of this specification, during the parallel execution of multiple threads, atomic locks are added to each community to avoid thread conflicts, and community attributes are obtained and updated in a timely manner. Changes in a vertex community will directly affect the calculation of subsequent vertices in this round of calculations, ensuring that the latest community attributes of the target community can be obtained for each entity vertex during calculations, speeding up the process of transmitting vertex state change information to other vertices, greatly improving the convergence speed, and reducing the convergence speed by more than an order of magnitude on some data sets, which can greatly improve timeliness. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification;

[0049] Figure 2 A flow chart of a community discovery method based on a structure graph according to an embodiment is shown;

[0050] Figure 3 A schematic diagram of an association diagram according to an embodiment is shown;

[0051] Figure 4A schematic block diagram of a community discovery apparatus based on a structure graph according to an embodiment is shown. DETAILED DESCRIPTION

[0052] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0053] Figure 1 This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves community discovery based on a structure diagram. It can be understood that a community usually includes multiple subjects, which can also be called entities. Community discovery is often used to identify group control subjects. In other words, the identification of group control subjects is a typical application scenario of community discovery.

[0054] The following is a brief description of the group control scenario. Machine traffic is the use of machine means to automatically implement certain operations and requests, such as logging in, collecting coupons, and clicking operations. The purposes of machine traffic include brushing, crawling, etc. Group control is the most serious type of machine traffic, usually referring to the centralized or distributed control of a group of intelligent agents or devices, guiding the overall behavior to achieve a predetermined goal under specific environments or conditions. For example, multiple mobile phones are controlled by one or more computers to automatically perform a series of tasks or operations. The group control system is completed through the collaboration of software and hardware. The hardware devices may include group control hosts, hubs, computer hosts, and terminal mobile phones. The software provides a sub-control function, allowing multiple mobile phones to be controlled in a local LAN environment. Among them, the aforementioned intelligent agents or devices each correspond to a corresponding user, and the identification of the group control subject is the identification of the user participating in the group control. The identification of the group control subject is a long-term capability in the field of traffic security, and has played a huge role in advertising, marketing, search and push, etc. In the embodiment of this specification, through community discovery based on the structure diagram, support is provided for the identification of group control groups composed of group control subjects, which is conducive to maintaining traffic security.

[0055] Reference Figure 1 , shows a typical example of a structure graph, which is a heterogeneous graph, including M entity vertices representing M users, and several auxiliary vertices representing the optional feature values ​​of N types of features of the users, and there is a connecting edge between any entity vertex and the auxiliary vertex corresponding to the actual feature value of the user. For example, Figure 1The entity vertices U1, U2, U3, and U4 represent users U1, U2, U3, and U4 respectively, and the auxiliary vertices A1, A2, B1, B2, and B3, wherein the auxiliary vertices A1 and A2 correspond to two optional feature values ​​of feature A respectively, and the auxiliary vertices B1, B2, and B3 correspond to three optional feature values ​​of feature B respectively. The actual feature value of feature A of user U1 corresponds to auxiliary vertex A1, and there is a connection edge between entity vertex U1 and auxiliary vertex A1; the actual feature value of feature B of user U1 corresponds to auxiliary vertex B1, and there is a connection edge between entity vertex U1 and auxiliary vertex B1; the actual feature value of feature A of user U2 corresponds to auxiliary vertex A2, and there is a connection edge between entity vertex U2 and auxiliary vertex A2; the actual feature value of feature B of user U2 corresponds to auxiliary vertex B2, and there is a connection edge between entity vertex U2 and auxiliary vertex B2; the actual feature value of feature A of user U3 corresponds to auxiliary vertex A1, and there is a connection edge between entity vertex U3 and auxiliary vertex A1; the actual feature value of feature B of user U3 corresponds to auxiliary vertex B3, and there is a connection edge between entity vertex U3 and auxiliary vertex B3; the actual feature value of feature A of user U4 corresponds to auxiliary vertex A2, and there is a connection edge between entity vertex U4 and auxiliary vertex A2; the actual feature value of feature B of user U4 corresponds to auxiliary vertex B2, and there is a connection edge between entity vertex U4 and auxiliary vertex B2.

[0056] In the embodiments of this specification, the structure graph may be a heterogeneous graph or a homogeneous graph, and the structure graph may be a directed graph or an undirected graph, and the connecting edges may have edge weights or may not have edge weights.

[0057] The following are some basic concepts about graphs.

[0058] Directed graph: A graph with directed edges, indicating that the relationship between vertices is directed, from one vertex to another.

[0059] Undirected graph: A graph with no direction on the edges, indicating that the relationship between vertices is bidirectional, or has no specific direction.

[0060] Edge weight: A numerical value assigned to an edge in a graph that represents some property of the edge, such as the cost or strength of traveling from one vertex to another.

[0061] Weighted graph: The edges in the graph are assigned edge weights, such as edge weights representing length, time, cost or capacity.

[0062] Unweighted graph: The edges in the graph have no edge weights, or all edges have the same edge weight, which only indicates the existence of a connection without conveying other information.

[0063] It should be noted that Figure 1As just an example of a structure graph, in practice the scale of the structure graph will be much larger, usually including more entity vertices and auxiliary vertices, for example, the number of vertices can reach hundreds of thousands or millions.

[0064] In the embodiment of this specification, when the structure graph is a heterogeneous graph, the vertices in the graph are divided into entity vertices and auxiliary vertices, and the entity vertices are divided into non-overlapping communities. The relationship between the entity vertices is not considered, that is, there is no association relationship within the community. In community discovery based on the structure graph, in order to significantly improve timeliness, a corresponding solution is provided.

[0065] Among them, the execution methods of various graph algorithms were studied and the final execution method was proposed. Common execution methods of graph algorithms include the following:

[0066] Serial computing: The computer can only perform one task or operation at a time. The computing task will not be divided, and each step must be started after the previous step is completed.

[0067] Parallel computing: The process of performing multiple tasks or operations simultaneously in a computer. In a multi-core processor, different cores can perform different tasks or parts of the same task simultaneously, speeding up the overall process.

[0068] Distributed: Computing tasks are distributed to different physical machines for execution. These machines are connected through a network to complete collaborative tasks together.

[0069] Synchronous computing: All threads or processes executing in parallel or distributed manner must wait at a specific synchronization point until all other threads or processes reach this point before they can continue to perform subsequent work.

[0070] Asynchronous computing: Parallel or distributed execution of threads or processes that allows tasks or computations to proceed freely at their own rates without requiring strict time alignment or synchronization with other tasks.

[0071] The embodiments of this specification provide a solution for asynchronous parallel graph computing, which can improve computing efficiency and the timeliness of result output in community discovery based on structure graphs.

[0072] Figure 2 A flowchart of a community discovery method based on a structure graph according to an embodiment is shown, wherein the structure graph includes M entity vertices representing M users. The method can be based on Figure 1 The implementation scenario shown in Figure 1 is as follows. Figure 2As shown, the community discovery method based on the structure graph in this embodiment includes the following steps: Step 21, based on the structure graph, initialize the community graph, wherein each community is represented by an entity vertex and has a community attribute, wherein the community attribute includes a deterministic index of the corresponding community; each community is added with an atomic lock; Step 22, multiple threads are used to perform several rounds of first graph iterations in parallel, wherein the current round of graph iterations performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update; Step 23, according to the community affiliation of the entity vertex after the several rounds of first graph iterations, aggregating the entity vertices in any community into a new entity vertex representing the community, so as to update the community graph. The specific execution method of each of the above steps is described below.

[0073] First, in step 21, based on the structure graph, a community graph is initialized, where each community is represented by an entity vertex and has community attributes, and the community attributes include a certainty index of the corresponding community; each community is added with an atomic lock. It can be understood that there can be multiple community attributes, and the certainty index is one of them. The certainty index can be used to measure the closeness of the community, so as to determine whether the entity vertex moves to join the community.

[0074] Among them, the atomic lock is a special lock used to protect concurrent access to shared data and ensure the atomicity of the operation, that is, only one thread can access the protected data at any time. During the calculation process, a community attribute may be used by multiple threads at the same time, and changes may cause read-write conflicts. Therefore, an atomic lock is needed to ensure that other threads do not access the attribute value that needs to be modified, and release the atomic lock after the modification is completed. Other threads will remain in a waiting state until the atomic lock is released before they can read and write this attribute value.

[0075] In the embodiment of this specification, one entity vertex is initialized as a community, each community only contains the entity vertex, and the number of communities is the same as the number of entity vertices. For example, Figure 1 In , four communities are initialized, and entity vertices U1, U2, U3, and U4 each correspond to a community.

[0076] In one example, the structural diagram also includes several auxiliary vertices representing the optional feature values ​​of each of the N types of features of the user. There is a connecting edge between any entity vertex and the auxiliary vertex corresponding to the actual feature value of the user, and the deterministic index is determined based on the distribution of feature values ​​of various types of auxiliary vertices associated with each entity vertex in the community.

[0077] In this example, the structural graph is a heterogeneous graph, which includes not only entity vertices but also auxiliary vertices. Two entity vertices can be connected through auxiliary vertices, and the distribution of feature values ​​of auxiliary vertices in the community should be tighter.

[0078] In one example, the entity vertex has a vertex attribute; the vertex attribute is used to indicate the community to which the corresponding entity vertex belongs.

[0079] In this example, the community to which the entity vertex belongs can be determined through the vertex attributes of the entity vertex.

[0080] In one example, the community graph is initialized based on the structure graph and is performed based on a parallel computing mode.

[0081] In this example, the process of initializing the community graph is also a parallel computing process, which can speed up its processing.

[0082] In one example, the method further includes:

[0083] Based on the community graph, an association graph is initialized; if a preset relationship is satisfied between a first entity vertex and a second entity vertex in the community graph, the first entity vertex and the second entity vertex are added to the association graph, and a connecting edge between the two is added to the association graph.

[0084] In this example, an association graph is constructed in advance, and all entity vertices that satisfy the preset relationship are added to the association graph. The local movement process can be accelerated through the association graph, and by adding restrictions to the association graph, that is, setting the above preset relationship, the community size can be further controlled and the calculation can be accelerated, so that small-scale communities can be detected more accurately. For example, two entity vertices are required to be connected to more than three identical auxiliary vertices before these two entity vertices are added to the association graph. It can be understood that the association graph only contains entity vertices, not auxiliary vertices. For the topological structure of the association graph, pruning and screening are performed in advance when processing the graph data of the community graph, which avoids the calculation of weakly associated entity vertices, further speeding up the calculation while greatly reducing the memory usage of the topological structure of the association graph.

[0085] Figure 3 FIG. 1 is a schematic diagram of an association diagram according to an embodiment. Figure 3 , assuming that the aforementioned preset relationship is specifically the second-degree neighbor entity vertex with which it has at least one commonly connected auxiliary vertex. Corresponding to Figure 1In a heterogeneous graph, entity vertices U1 and U3 have a common connected auxiliary vertex A1. Entity vertices U1 and U3 are added to the association graph, and a connecting edge between them is added to the association graph; entity vertices U2 and U4 have a common connected auxiliary vertex A2. Entity vertices U2 and U4 are added to the association graph, and a connecting edge between them is added to the association graph. It should be noted that Figure 3 As an example of an association graph, the scale of an association graph in practice is much larger, and usually contains more entity vertices, for example, the number of vertices reaches hundreds of thousands or millions. In an association graph, the number of other entity vertices that have an association relationship with a certain entity vertex can be one or more.

[0086] In one example, the community graph or the association graph uses compressed row storage (compressed sparse row, CSR) or compressed column storage (compressed sparse column, CSC) to store graph structure information.

[0087] In this example, a compressed CSR or CSC structure is used for the graph structure of the community graph or association graph, so that the graph topology structure is more compact and the memory usage of the graph topology structure is reduced.

[0088] Furthermore, the community attributes also include:

[0089] A set of feature values ​​of various auxiliary vertices associated with each entity vertex in the community; and / or,

[0090] The number of entity vertices contained in the community.

[0091] In this example, the community attribute may include other information in addition to the deterministic index, which may be used for graph calculation in community discovery, for example, calculating the deterministic index after the entity vertex is moved, or displaying it as the final result.

[0092] Then in step 22, multiple threads are used to perform several rounds of first graph iterations in parallel, wherein the current round of graph iterations performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining the target communities to which each neighbor entity vertex that satisfies a preset relationship belongs, and after obtaining the atomic locks of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update. It can be understood that the above movement refers to the first entity vertex moving from the cell to which it originally belongs to a target community, and the community attributes of the target community will change.

[0093] In one example, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs includes:

[0094] Traverse each neighbor entity vertex that satisfies the preset relationship with it, and determine the community where the neighbor entity vertex is located according to the vertex attributes associated with the neighbor entity vertex.

[0095] In this example, each vertex has a vertex attribute that identifies the community it belongs to.

[0096] Furthermore, the method further comprises:

[0097] When movement is required, the vertex attributes of the first entity vertices are updated.

[0098] In this example, the vertex attributes of the entity vertices are updated in real time.

[0099] In one example, determining whether the first entity vertex needs to be moved according to the community attributes of each target community includes:

[0100] According to the community attribute of any target community, determining the gain of the deterministic index after moving the first entity vertex to the target community;

[0101] According to the gains of the target communities, it is determined whether the first entity vertex needs to be moved to one of the target communities.

[0102] In this example, the community attributes include the certainty index of the corresponding community. When the first entity vertex moves to the target community, the certainty index of the target community will change. By calculating the gain of the certainty index after moving the first entity vertex to the target community, it is determined whether the first entity vertex needs to be moved to one of the target communities. This can ensure that the indicator value of the certainty index of the community finally divided is larger, that is, the community has a higher degree of compactness.

[0103] In one example, the preset relationship includes: a second-degree neighbor entity vertex with which it has at least one auxiliary vertex connected in common; or a second-degree neighbor entity vertex with which it has multiple auxiliary vertices connected in common.

[0104] In this example, the target communities are screened by using the above preset relationships, thereby avoiding the calculation of weakly associated target communities and further speeding up the calculation.

[0105] In one example, before determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, the method further includes:

[0106] In the association graph, entity vertices having connecting edges therewith are searched as neighbor entity vertices that satisfy a preset relationship therewith.

[0107] In this example, the association graph only includes entity vertices and the association relationships between entity vertices. It is more convenient and faster to search for each neighbor entity vertex in the association graph than to search for each neighbor entity vertex in the community graph.

[0108] In one example, determining a gain in a certainty indicator after moving the first entity vertex to the target community includes:

[0109] Adding the feature values ​​of various auxiliary vertices associated with the first entity vertex to the set of feature values ​​of the target community to obtain an updated set;

[0110] Calculating an updated certainty index according to the update set;

[0111] The gain is obtained by subtracting the certainty index before updating included in the community attribute from the certainty index after updating.

[0112] In this example, the updated certainty index is calculated by updating the set of feature values ​​of the target community, thereby obtaining the above gain.

[0113] In one example, the stopping condition of the first graph iteration is that the communities to which each entity vertex belongs no longer change.

[0114] In this example, after a round of iterative calculation is completed, it is determined whether the community to which the entity vertex belongs has changed. If so, it proves that the calculation result has not converged, and the next round of iteration is continued; if the community to which the entity vertex belongs has not changed, it proves that the calculation result has converged, and the first graph iteration is stopped.

[0115] Finally, in step 23, according to the community affiliation of the entity vertices after the first graph iterations, the entity vertices in any community are aggregated into new entity vertices representing the community to update the community graph. It can be understood that the graph topology of the community graph will change after the above aggregation.

[0116] In the embodiments of the present specification, the above-mentioned aggregation can be performed according to vertex attributes to update the graph topology of the community graph, and optionally, the graph topology of the association graph can also be updated. Among them, the entity vertices in the updated community graph represent the communities after several rounds of iterations of the first graph, and the edges are the edges from the entity vertices contained in the community to each auxiliary vertex before aggregation, and the edge weights are the sum of the edge weights of the community pointing to the same auxiliary vertex before aggregation; the entity vertices in the updated association graph represent the communities after several rounds of iterations of the first graph, and the edges are the edges from the entity vertices contained in the community to other associated entity vertices before aggregation, and duplicate edges and self-loop edges can be removed as needed. At the same time, in the process of updating the graph topology, topology change operations can be performed on the original topology, which reduces unnecessary graph structure and attribute changes, speeds up the topology update process, and avoids the memory expansion problem caused by the coexistence of new and old topologies.

[0117] In one example, the method further includes:

[0118] For the updated community graph, several rounds of second graph iterations are performed, wherein at least one round of second graph iterations includes: using multiple threads to execute the several rounds of first graph iterations in parallel.

[0119] In this example, the second graph iteration is included in the first graph iteration. The graph topology structure of the community graph targeted by the second graph iterations of different rounds has changed, while in a certain round of the second graph iteration, the graph topology structure of the community graph targeted by the first graph iterations of different rounds has not changed.

[0120] Furthermore, the several rounds of second graph iterations form multi-level communities, and a hierarchical relationship is formed between the community obtained by the N+1th round of second graph iteration and the community obtained by the Nth round of second graph iteration.

[0121] For example, a community obtained in the Nth round of graph iteration includes entity vertices U1 and U2, and a community obtained in the N+1th round of graph iteration includes entity vertices U1, U2, and U4, and a hierarchical relationship is formed between the two communities.

[0122] Furthermore, after performing several rounds of second graph iterations, the method further includes:

[0123] Output the community attributes corresponding to each community.

[0124] In this example, not only the final community division result but also the community attributes corresponding to each community may be output.

[0125] Furthermore, the stopping condition of the second graph iteration is that the number of communities no longer changes.

[0126] In this example, after updating the community graph, it is possible to determine whether the number of communities has changed, that is, to determine whether the graph topology of the community graph has changed. If it has changed, it proves that the multi-level calculation process is not completed and it is necessary to continue to execute the second graph iteration; if it has not changed, it proves that the multi-level calculation process is completed and the second graph iteration can be stopped.

[0127] In the embodiments of this specification, the first graph iteration, aggregation, community graph initialization and update steps in the processing process can all adopt parallel computing processes. For these parallel computing processes, the task stealing thread scheduling method can be adopted to ensure that each thread can steal the tasks of other threads that have not completed tasks on the premise of completing the tasks of the thread first, and ensure that the final task volume of all threads is basically the same, realizing thread-level load balancing, avoiding the waste of computing resources, and further speeding up the computing process.

[0128] According to the method provided by the embodiments of the present specification, a community graph is first initialized based on a structure graph, wherein each community is represented by an entity vertex and has community attributes, wherein the community attributes include a deterministic index of the corresponding community; an atomic lock is added to each community; and then multiple threads are used to perform several rounds of first graph iterations in parallel, wherein the current round of graph iterations performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update; finally, according to the community affiliation of the entity vertex after the several rounds of first graph iterations, the entity vertices in any community are aggregated into a new entity vertex representing the community to update the community graph. As can be seen from the above, in the embodiments of this specification, during the parallel execution of multiple threads, atomic locks are added to each community to avoid thread conflicts, and community attributes are obtained and updated in a timely manner. Changes in a vertex community will directly affect the calculation of subsequent vertices in this round of calculations, ensuring that the latest community attributes of the target community can be obtained for each entity vertex during calculations, speeding up the process of transmitting vertex state change information to other vertices, greatly improving the convergence speed, and reducing the convergence speed by more than an order of magnitude on some data sets, which can greatly improve timeliness.

[0129] In addition, the embodiments of this specification simplify the graph topology structure update process and reduce the memory usage of the graph topology structure through compact graph structure, local update of graph topology, and pruning of associated graph topology, while avoiding the memory expansion problem caused by the coexistence of new and old topological structures, and have the advantage of low memory usage. Through task stealing thread scheduling, thread-level load balancing is achieved, computing resources can be fully utilized, and the consistency of the computing amount of each thread is guaranteed. Through the above-mentioned low memory usage, the calculation process can be executed asynchronously and in parallel in stand-alone mode, and at the same time, with the help of convergence acceleration and thread load balancing of asynchronous computing, the time overhead is greatly reduced.

[0130] According to another aspect of the embodiment, a community discovery device based on a structure graph is also provided, wherein the structure graph includes M entity vertices representing M users, and the device is used to execute the method provided in the embodiment of this specification. Figure 4 FIG. 1 is a schematic block diagram of a community discovery apparatus based on a structure diagram according to an embodiment. Figure 4 As shown, the device 400 includes:

[0131] An initialization unit 41 is used to initialize a community graph based on the structure graph, wherein each community is represented by an entity vertex and has community attributes, and the community attributes include a certainty index of the corresponding community; each community is added with an atomic lock;

[0132] The first iteration unit 42 is used to use multiple threads to perform several rounds of first graph iteration in parallel, wherein the current round of graph iteration performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update;

[0133] The aggregation unit 43 is used to aggregate the entity vertices in any community into new entity vertices representing the community according to the community affiliation of the entity vertices after the first iteration unit 42 performs several rounds of first graph iterations, so as to update the community graph.

[0134] Optionally, as an embodiment, the structure diagram also includes a number of auxiliary vertices representing the optional feature values ​​of each of the N types of features of the user, and there is a connecting edge between any entity vertex and the auxiliary vertex corresponding to the actual feature value of the user, and the deterministic index is determined based on the distribution of feature values ​​of various types of auxiliary vertices associated with each entity vertex in the community.

[0135] Optionally, as an embodiment, the entity vertex has vertex attributes; the vertex attributes are used to indicate the community to which the corresponding entity vertex belongs; the first iteration unit 42 is specifically used to traverse each neighbor entity vertex that satisfies a preset relationship with it, and determine the community to which it belongs based on the vertex attributes associated with the neighbor entity vertex.

[0136] Furthermore, the first iteration unit 42 is further configured to update vertex attributes of the first entity vertices when movement is required.

[0137] Optionally, as an embodiment, the first iteration unit 42 includes:

[0138] a gain determination subunit, configured to determine, according to a community attribute of any target community, a gain of a certainty indicator after moving the first entity vertex to the target community;

[0139] The movement determination subunit is used to determine whether the first entity vertex needs to be moved to one of the target communities according to the gains of each target community obtained by the gain determination subunit.

[0140] Optionally, as an embodiment, the device further includes:

[0141] The second iteration unit is used to perform several rounds of second graph iterations on the updated community graph obtained by the aggregation unit 43, wherein at least one round of the second graph iteration includes: using multiple threads to perform the several rounds of first graph iterations in parallel.

[0142] Optionally, as an embodiment, the initialization unit 41 initializes the community graph based on the structure graph, based on a parallel computing mode.

[0143] Optionally, as an embodiment, the preset relationship includes: a second-degree neighbor entity vertex with which it has at least one auxiliary vertex connected in common; or a second-degree neighbor entity vertex with which it has multiple auxiliary vertices connected in common.

[0144] Optionally, as an embodiment, the initialization unit 41 is further used to initialize an association graph based on the community graph; wherein if the first entity vertex and the second entity vertex in the community graph satisfy the preset relationship, the first entity vertex and the second entity vertex are added to the association graph, and a connecting edge between the two is added to the association graph;

[0145] The first iteration unit 42 is further configured to search, in the association graph, for entity vertices having connecting edges with the target communities to which the neighbor entity vertices satisfying the preset relationship belong, before determining the target communities to which the neighbor entity vertices satisfying the preset relationship belong.

[0146] Furthermore, the community graph or the association graph adopts a storage method of row compression storage CSR or column compression storage CSC to store graph structure information.

[0147] Furthermore, the several rounds of second graph iterations form multi-level communities, and a hierarchical relationship is formed between the community obtained by the N+1th round of second graph iteration and the community obtained by the Nth round of second graph iteration.

[0148] Furthermore, the community attributes also include:

[0149] The set of feature values ​​of various auxiliary vertices associated with each entity vertex in the community;

[0150] The gain determination subunit comprises:

[0151] A set updating module, used for adding the feature values ​​of various auxiliary vertices associated with the first entity vertex to the set of feature values ​​of the target community to obtain an updated set;

[0152] An indicator updating module, used for calculating an updated certainty indicator according to the updated set obtained by the set updating module;

[0153] The determination module is used to subtract the certainty index before updating included in the community attribute from the updated certainty index obtained by the index updating module to obtain the gain.

[0154] Furthermore, the community attributes also include:

[0155] The number of entity vertices contained in the community;

[0156] The device also includes:

[0157] The output unit is used to output the community attributes corresponding to each community after executing several rounds of second graph iterations.

[0158] Optionally, as an embodiment, the stopping condition of the first graph iteration is that the communities to which each entity vertex belongs no longer change.

[0159] Furthermore, the stopping condition of the second graph iteration is that the number of communities no longer changes.

[0160] Through the device provided by the embodiment of the present specification, first, the initialization unit 41 initializes the community graph based on the structure graph, wherein each community is represented by an entity vertex and has community attributes, and the community attributes include a deterministic indicator of the corresponding community; each community is added with an atomic lock; then the first iteration unit 42 uses multiple threads to perform several rounds of first graph iteration in parallel, wherein the current round of graph iteration performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and when it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update; finally, the aggregation unit 43 aggregates the entity vertices in any community into a new entity vertex representing the community according to the community affiliation of the entity vertices after the several rounds of first graph iterations, so as to update the community graph. As can be seen from the above, in the embodiments of this specification, during the parallel execution of multiple threads, atomic locks are added to each community to avoid thread conflicts, and community attributes are obtained and updated in a timely manner. Changes in a vertex community will directly affect the calculation of subsequent vertices in this round of calculations, ensuring that the latest community attributes of the target community can be obtained for each entity vertex during calculations, speeding up the process of transmitting vertex state change information to other vertices, greatly improving the convergence speed, and reducing the convergence speed by more than an order of magnitude on some data sets, which can greatly improve timeliness.

[0161] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.

[0162] According to another embodiment of the present invention, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the Figure 2 The method described.

[0163] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the present invention can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0164] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A community discovery method based on a structure graph, wherein the structure graph includes M entity vertices representing M users, the method comprising: Initializing a community graph based on the structure graph, wherein each community is represented by an entity vertex and has community attributes, wherein the community attributes include a certainty indicator of the corresponding community; Atomic locks are added to each community; A plurality of threads are used to execute several rounds of first graph iterations in parallel, wherein a current round of graph iteration executed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining the atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update; According to the community affiliations of the entity vertices after the several rounds of first graph iterations, the entity vertices in any community are aggregated into new entity vertices representing the community, so as to update the community graph.

2. The method of claim 1, wherein: The structural diagram also includes several auxiliary vertices representing the optional feature values ​​of each of the N types of features of the user. There is a connecting edge between any entity vertex and the auxiliary vertex corresponding to the actual feature value of the user. The deterministic index is determined based on the distribution of feature values ​​of various types of auxiliary vertices associated with each entity vertex in the community.

3. The method of claim 1, wherein: The entity vertex has a vertex attribute; the vertex attribute is used to indicate the community to which the corresponding entity vertex belongs; The step of determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs includes: Traverse each neighbor entity vertex that satisfies the preset relationship with it, and determine the community where the neighbor entity vertex is located according to the vertex attributes associated with the neighbor entity vertex.

4. The method of claim 3, wherein: The method further comprises: When movement is required, the vertex attributes of the first entity vertices are updated.

5. The method of claim 1, wherein: The determining whether the first entity vertex needs to be moved according to the community attributes of each target community includes: According to the community attribute of any target community, determining the gain of the deterministic index after moving the first entity vertex to the target community; According to the gains of the target communities, it is determined whether the first entity vertex needs to be moved to one of the target communities.

6. The method of claim 1, wherein: The method further comprises: For the updated community graph, several rounds of second graph iterations are performed, wherein at least one round of second graph iterations includes: using multiple threads to execute the several rounds of first graph iterations in parallel.

7. The method of claim 1, wherein: The community graph is initialized based on the structure graph and is performed based on a parallel computing mode.

8. The method of claim 1, wherein: The preset relationship includes: a second-degree neighbor entity vertex with which it has at least one auxiliary vertex connected in common; or a second-degree neighbor entity vertex with which it has multiple auxiliary vertices connected in common.

9. The method of claim 1, wherein: The method further comprises: Based on the community graph, an association graph is initialized; wherein if the first entity vertex and the second entity vertex in the community graph satisfy the preset relationship, the first entity vertex and the second entity vertex are added to the association graph, and a connecting edge between the two is added to the association graph; Before determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, the method further includes: In the association graph, entity vertices having connecting edges therewith are searched as neighbor entity vertices that satisfy a preset relationship therewith.

10. The method of claim 9, wherein: The community graph or the association graph adopts a storage method of row compression storage CSR or column compression storage CSC to store graph structure information.

11. The method of claim 6, wherein: The several rounds of second graph iteration form multi-level communities, and a hierarchical relationship is formed between the community obtained by the N+1th round of second graph iteration and the community obtained by the Nth round of second graph iteration.

12. The method of claim 5, wherein: The community attributes also include: The set of feature values ​​of various auxiliary vertices associated with each entity vertex in the community; The determining a gain of a certainty indicator after moving the first entity vertex to the target community includes: Adding the feature values ​​of various auxiliary vertices associated with the first entity vertex to the set of feature values ​​of the target community to obtain an updated set; Calculating an updated certainty index according to the update set; The gain is obtained by subtracting the certainty index before updating included in the community attribute from the certainty index after updating.

13. The method of claim 6, wherein: The community attributes also include: The number of entity vertices contained in the community; After performing several rounds of second graph iterations, the method further includes: Output the community attributes corresponding to each community.

14. The method of claim 1, wherein: The stopping condition of the first graph iteration is that the communities to which each entity vertex belongs no longer change.

15. The method of claim 6, wherein: The stopping condition for the second graph iteration is that the number of communities no longer changes.

16. A community discovery device based on a structure graph, wherein the structure graph includes M entity vertices representing M users, the device comprising: An initialization unit, configured to initialize a community graph based on the structure graph, wherein each community is represented by an entity vertex and has community attributes, wherein the community attributes include a certainty index of the corresponding community; Atomic locks are added to each community; A first iteration unit is configured to use multiple threads to perform several rounds of first graph iterations in parallel, wherein a current round of graph iterations performed by any first thread includes, for any first entity vertex that the first thread is responsible for processing, determining each target community to which each neighbor entity vertex that satisfies a preset relationship belongs, after obtaining an atomic lock of each target community, determining whether the first entity vertex needs to be moved according to the community attributes of each target community, and if it needs to be moved, updating the community attributes of the relevant community, and releasing the atomic lock after the update; The aggregation unit is used to aggregate entity vertices in any community into new entity vertices representing the community according to the community affiliation of the entity vertices after the first iteration unit performs several rounds of first graph iterations, so as to update the community graph.

17. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 15.

18. A computing device, comprising a memory and a processor, wherein the memory stores executable codes, and when the processor executes the executable codes, the method according to any one of claims 1 to 15 is implemented.

Images