Method, device, equipment and storage medium for mining strongly connected components of graph data

Abstract

The present application discloses a method, device, equipment and storage medium for mining strongly connected components of graph data. The method obtains a directed graph including multiple nodes; then divides the directed graph into multiple block graphs, and performs asynchronous label propagation update on the first labels of the nodes in each block graph, thereby greatly accelerating the speed of first label iteration, and after the first label update of all nodes in the directed graph is completed, the nodes with the same first labels before and after the update are used as the first child nodes, and the nodes in the directed graph that belong to the same strongly connected component as the first child node can be determined; the method can accelerate the update speed of the first label through asynchronous label propagation update, thereby reducing the number of iterations required in the label propagation process, effectively improving the efficiency of strongly connected component mining, and helping to reduce the consumption of computing resources. The present application can be widely used in the field of big data technology.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a method, device, equipment and storage medium for mining strongly connected components of graph data. Background Art

[0002] As a data structure, graphs are widely used in various fields of natural sciences and social sciences. For example, the connectivity of nodes in a graph is often considered, analyzed, and applied in scenarios such as telecommunications and power networks, transportation systems, computer programming, fault diagnosis, artificial intelligence, printed circuit board design, and user analysis.

[0003] Currently, research on node connectivity issues includes the task of mining strongly connected components in directed graphs. The mining algorithms used in related technologies are limited by the amount of graph data when executed on a single machine, and are difficult to adapt to large-scale graph data needs; while the algorithms using distributed mining often require too many iterations and consume a lot of hardware resources. In summary, the method of mining strongly connected components of graph data used in related technologies is not very practical, and the mining effect is often not ideal. Summary of the invention

[0004] The purpose of this application is to solve one of the technical problems existing in the prior art to at least a certain extent.

[0005] To this end, one purpose of an embodiment of the present application is to provide a method for mining strongly connected components of graph data, which can improve the efficiency of mining strongly connected components and help reduce the consumption of computing resources.

[0006] In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present application include:

[0007] On the one hand, an embodiment of the present application provides a method for mining strongly connected components of graph data, the method comprising the following steps:

[0008] Obtain a directed graph; the directed graph includes a plurality of nodes, each of the nodes corresponds to a first label, the first label includes a first in-degree attribute value and a first out-degree attribute value; different nodes correspond to different first in-degree attribute values, and different nodes correspond to different first out-degree attribute values;

[0009] Dividing the directed graph into a plurality of block graphs;

[0010] The first label of each node in the directed graph is stored, the first label of each node in the block graph is asynchronously updated by label propagation according to the stored first label, and the stored first label is updated according to the result of the label propagation update;

[0011] After all nodes in the directed graph are updated, it is determined that the nodes carrying the same first label as the first child node belong to the same strongly connected component; the first child node is a node with the same first label before and after the update.

[0012] On the other hand, an embodiment of the present application provides a device for mining strongly connected components of graph data, the device comprising:

[0013] An acquisition module, used for acquiring a directed graph; the directed graph includes a plurality of nodes, each of the nodes corresponds to a first label, the first label includes a first in-degree attribute value and a first out-degree attribute value; different nodes correspond to different first in-degree attribute values, and different nodes correspond to different first out-degree attribute values;

[0014] A partitioning module, used for partitioning the directed graph into a plurality of block graphs;

[0015] A storage and update module, configured to store the first label of each node in the directed graph, perform asynchronous label propagation update on the first label of each node in the block graph according to the stored first label, and update the stored first label according to the result of the label propagation update;

[0016] The processing module is used to determine that the nodes carrying the same first label as the first child node belong to the same strongly connected component after all nodes in the directed graph are updated; the first child node is a node with the same first label before and after the update.

[0017] Optionally, in some embodiments, the device further comprises:

[0018] A bidirectional edge compression module is used to determine a node pair in the directed graph and merge the node pair into one node; the node pair includes a second child node and a third child node, and the second child node and the third child node have a bidirectional connection relationship.

[0019] Optionally, in some embodiments, the device further comprises:

[0020] a propagation module, configured to perform label propagation update on the first in-degree attribute value and the first out-degree attribute value of the node in the directed graph to obtain a second label of each node; the second label includes a second in-degree attribute value and a second out-degree attribute value;

[0021] A decoupling module is used to determine and delete a decoupling edge in the directed graph according to the second label; the nodes connected at both ends of the decoupling edge correspond to different second labels.

[0022] Optionally, in some embodiments, the storage and update module includes:

[0023] A first storage module, configured to store the first in-degree attribute value of each of the nodes in the directed graph to a server;

[0024] A first pulling module is used to asynchronously pull the first in-degree attribute value of the nodes in each of the block graphs and the third in-degree attribute value corresponding to each of the nodes through multiple processing units; wherein each of the processing units processes at least one of the block graphs; the third in-degree attribute value is the first in-degree attribute value of the in-degree connection node of the node;

[0025] The first updating module is used to update the first in-degree attribute value of the node in the processing unit according to the third in-degree attribute value corresponding to the node, and synchronize the updated first in-degree attribute value to the server.

[0026] Optionally, in some embodiments, the first update module includes:

[0027] A first determining submodule, configured to determine a minimum value or a maximum value between the first in-degree attribute value and the third in-degree attribute value corresponding to the node;

[0028] The first processing submodule is used to use the minimum value or the maximum value as the updated first in-degree attribute value of the node.

[0029] Optionally, in some embodiments, the storage and update module includes:

[0030] A second storage module, configured to store the first out-degree attribute value of each of the nodes in the directed graph to a server;

[0031] A second pulling module is used to asynchronously pull the first out-degree attribute value of the nodes in each of the block graphs and the third out-degree attribute value corresponding to each of the nodes through multiple processing units; wherein each of the processing units processes at least one of the block graphs; the third out-degree attribute value is the first out-degree attribute value of the out-degree connection node of the node;

[0032] The second updating module is used to update the first out-degree attribute value of the node in the processing unit by the third out-degree attribute value corresponding to the node, and synchronize the updated first out-degree attribute value to the server.

[0033] Optionally, in some embodiments, the second update module includes:

[0034] A second determining submodule, configured to determine a minimum value or a maximum value between the first out-degree attribute value and the third out-degree attribute value corresponding to the node;

[0035] The second processing submodule uses the minimum value or the maximum value as the updated first out-degree attribute value of the node.

[0036] Optionally, in some embodiments, the device further comprises:

[0037] The extraction module is used to extract the determined strongly connected components from the directed graph to obtain an updated directed graph.

[0038] Optionally, in some embodiments, the device further comprises:

[0039] A first detection submodule, used to detect the number of nodes in the directed graph;

[0040] A third processing submodule is configured to send the directed graph to a single processing unit for strongly connected component mining when the number of nodes is less than or equal to a first threshold; or

[0041] A second detection submodule is used to detect the number of connection relationships between nodes in the directed graph;

[0042] The fourth processing submodule is used to send the directed graph to a single processing unit for strongly connected component mining when the number of connection relationships is less than or equal to a second threshold.

[0043] Optionally, in some embodiments, the device further comprises:

[0044] An initialization module is used to initialize the first label corresponding to each node in the updated directed graph when the number of nodes is greater than a first threshold or when the number of connection relationships is greater than a second threshold, and return to the partitioning module to execute the step of partitioning the directed graph into multiple block graphs.

[0045] Optionally, in some embodiments, the third processing submodule or the fourth processing submodule includes:

[0046] A sending submodule, used for sending the directed graph to a single processing unit;

[0047] A first search submodule, configured to perform a depth-first search on the directed graph in the processing unit, and add each of the nodes to a stack in the order in which they are searched;

[0048] A second search submodule is used to search for a fourth subnode from the top of the stack to the middle of the stack in sequence; the first timestamp and the second timestamp corresponding to the fourth subnode are the same; wherein the first timestamp is the timestamp when each of the nodes is searched, and the second timestamp is the earliest timestamp that can be traced back to each of the nodes;

[0049] The mining submodule is used to determine that the nodes having the same first timestamp or second timestamp as the fourth subnode belong to the same strongly connected component.

[0050] Optionally, in some embodiments, the processing module includes:

[0051] A fifth processing submodule, configured to determine the first child node from the directed graph after all nodes in the directed graph are updated;

[0052] The third search submodule is used to perform a depth-first search starting from the first child node along the direction of the in-degree or out-degree to determine that the nodes carrying the same first label as the first child node belong to the same strongly connected component.

[0053] On the other hand, an embodiment of the present application provides a computer device, the device comprising:

[0054] at least one processor;

[0055] at least one memory for storing at least one program;

[0056] When the at least one program is executed by the at least one processor, the at least one processor implements the aforementioned method for mining strongly connected components of graph data.

[0057] On the other hand, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor. The aforementioned processor-executable program is used to implement the aforementioned method for mining strongly connected components of graph data when executed by the processor.

[0058] On the other hand, an embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions, and the computer instructions are stored in the computer-readable storage medium mentioned above; the processor of the computer device mentioned above can read the computer instructions from the computer-readable storage medium mentioned above, and the processor executes the computer instructions, so that the computer device executes the method for mining strongly connected components of graph data mentioned above.

[0059] The advantages and beneficial effects of the present invention will be partly given in the following description, partly become apparent from the following description, or be understood through the practice of the present invention:

[0060] The method for mining strongly connected components of graph data provided in the embodiments of the present application obtains a directed graph including a plurality of nodes, wherein each node corresponds to a first label, and the first label includes a first in-degree attribute value and a first out-degree attribute value of the node; then the directed graph is divided into a plurality of block graphs, and the first labels of the nodes in each block graph are asynchronously propagated and updated, thereby greatly accelerating the speed of first label iteration, and after the first labels of all nodes in the directed graph are updated, the nodes with the same first labels before and after the update are taken as the first child nodes, and the nodes in the directed graph that belong to the same strongly connected component as the first child node can be determined; the method for mining strongly connected components of graph data can accelerate the update speed of the first label through asynchronous label propagation updates, thereby reducing the number of iterations required in the label propagation process, effectively improving the efficiency of strongly connected component mining, and helping to reduce the consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A schematic diagram of an application scenario of a strongly connected component provided in an embodiment of the present application;

[0063] Figure 2 A schematic diagram of another application scenario of a strongly connected component provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of a process for mining strongly connected components of graph data provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of an obtained directed graph provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of dividing a block graph in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of dividing a block graph in another method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of updating a first in-degree attribute value in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0069] Figure 8 This is a schematic diagram of an updated first label of a directed graph in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0070] Fig. 9 A schematic diagram of an implementation environment for asynchronous updating in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0071] Fig.10 A schematic diagram of an implementation environment for asynchronous updating in another method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0072] Fig.11 A schematic diagram of the principle of asynchronously updating the first label in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0073] Fig.12 A schematic diagram of mining by a single processing unit in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0074] Fig.13 A schematic diagram of bidirectional edge compression in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0075] Fig.14 A schematic diagram of bilateral decoupling in a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0076] Fig.15 A schematic diagram of a processing flow of a method for mining strongly connected components of graph data provided in an embodiment of the present application;

[0077] Fig.16 A schematic diagram of the structure of a device for mining strongly connected components of graph data provided in an embodiment of the present application;

[0078] Fig.17 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0079] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0080] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0081] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are first described. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0082] Graph: refers to a data form composed of many nodes (also called vertices) connected to each other. The nodes can be entities such as people and organizations, and the connections between nodes (called edges) represent certain relationships (such as friendships, subordinate relationships, etc.); a graph can have only one type of node and one type of edge (called a single graph), or it can have multiple types of nodes or multiple types of edges (called a heterogeneous graph), where the edges in the graph can be directed edges (called a directed graph) or undirected edges (called an undirected graph).

[0083] Strong connectivity: For any two nodes u and v in a directed graph, if there is a directed path from node u to node v and a directed path from node v to node u, then node u and node v are said to be strongly connected.

[0084] Strongly connected graph: If any two nodes in a directed graph are strongly connected, then the directed graph is a strongly connected graph.

[0085] Strongly Connected Component (SCC): A maximal strongly connected subgraph of a directed graph is called its strongly connected component.

[0086] Depth-first search (DFS): A search algorithm that starts from a node v in the graph and executes the following steps: visit vertex v; start from the unvisited adjacent nodes of node v in turn and perform a depth-first traversal of the graph; until all nodes in the graph that have paths connecting to node v are visited; if there are still unvisited nodes in the graph, start from an unvisited node and perform a depth-first traversal again until all nodes in the graph have been visited.

[0087] The graph data strongly connected component mining method provided in the embodiments of the present application can be applied to big data technology. Big data refers to a collection of data that cannot be captured, managed and processed by conventional software tools within a certain time frame. It is a massive, high-growth and diversified information asset that requires a new processing model to have stronger decision-making power, insight discovery and process optimization capabilities. With the advent of the cloud era, big data has also attracted more and more attention. Big data requires special technologies to effectively process extremely large amounts of data. Technologies suitable for big data include large-scale parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the Internet and scalable storage systems.

[0088] The method for mining strongly connected components of graph data in the embodiment of the present application can apply data mining, distributed databases and cloud computing platforms in big data technology to process related graph data. For example, based on cloud computing technology, the mining task of strongly connected components is carried out in a distributed manner. Cloud computing is an emerging computing model that can distribute computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. The resource pool of cloud computing mainly includes computing devices (virtualized machines, including operating systems), storage devices and network devices.

[0089] With the development of computer technology, data analysis technology has gradually emerged and has been applied in all walks of life. Among them, graphs, as a typical data structure, are often used in various application scenarios. The analysis and development of graph data is also a hot field at present. In related technologies, the connectivity problem of graph data is an important part of its analysis and application. For a directed graph, such as a directed graph G, if for any two nodes in graph G (denoted as node u and node v), there is a path from node u to node v, and there is a path from node v to node u, then the directed graph can be called a strongly connected directed graph. For a non-strongly connected directed graph, its maximal strongly connected subgraph is called a strongly connected component.

[0090] The strongly connected components of directed graphs have a wide range of application scenarios: for example, in social software, it is often necessary to analyze the relationship network of people to determine the association between users. At this time, it may be necessary to determine the strongly connected components of the graph data composed of a group of user groups. Figure 1 , Figure 1In the example, user 11 has sent information to user 12, user 12 has sent information to user 13, and user 13 has sent information to user 11. Therefore, any two of user 11, user 12, and user 13 can transmit information through a third party, indicating that user 11, user 12, and user 13 are strongly related. Figure 2 In order to determine whether two communication devices can transmit information to each other in a communication network, it is necessary to determine the strongly connected components of the graph data composed of a group of communication devices. If two communication devices belong to the same strongly connected component, they can transmit information to each other. Similarly, the strongly connected components of directed graphs can also be used in transportation, reliability analysis, circuit fault diagnosis, intelligence analysis and other scenarios.

[0091] The application basis of the above-mentioned strongly connected components includes correctly mining strongly connected components from directed graphs. In the related technologies, the mining of strongly connected components mainly adopts an algorithm based on depth-first search, which can be specifically divided into a single-machine execution algorithm and a distributed execution algorithm. Among them, the single-machine execution mining algorithm has a large limitation on the amount of graph data that can be processed. When the amount of data is small, it can complete the mining of strongly connected components more efficiently, but it is difficult to adapt to the needs of large-scale graph data mining; although the distributed execution algorithm can complete the mining of strongly connected components of large batches of graph data, this mining method will result in a large amount of redundant calculations, the algorithm iteration speed is slow, it requires a lot of hardware resources, and the mining time is also relatively long. From the above, it can be seen that the strong connected component mining technology in the related technology, the single-machine execution algorithm is difficult to meet the actual application needs, even the distributed execution algorithm takes a long time, and the redundant calculation will consume a lot of computing resources, and the benefits are low.

[0092] In view of this, an embodiment of the present application provides a method for mining strongly connected components of graph data, which obtains a directed graph including multiple nodes, wherein each node corresponds to a first label, and the first label includes a first in-degree attribute value and a first out-degree attribute value of the node; then the directed graph is divided into multiple block graphs, and the first labels of the nodes in each block graph are asynchronously propagated and updated, thereby greatly accelerating the speed of first label iteration, and after the first labels of all nodes in the directed graph are updated, the nodes with the same first labels before and after the update are taken as the first child nodes, and the nodes in the directed graph that belong to the same strongly connected component as the first child node can be determined; the method for mining strongly connected components of graph data can accelerate the update speed of the first label through asynchronous label propagation updates, thereby reducing the number of iterations required in the label propagation process, effectively improving the efficiency of strongly connected component mining, and helping to reduce the consumption of computing resources.

[0093] The specific implementation of the embodiment of the present application will be described below in conjunction with the accompanying drawings. First, the method for mining strongly connected components of graph data in the embodiment of the present application will be described.

[0094] In an embodiment of the present application, a method for mining strongly connected components of graph data is provided, which can be independently applied to a terminal or server, or can be executed based on data interaction between a terminal and a server, and can also be applied to software in a terminal or server to implement a portion of the software functions. In some embodiments, the above-mentioned terminal can be a smart phone, a tablet computer, a laptop computer, or a desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, and big data and artificial intelligence platforms; the software can be an independent application or a small program that depends on a host program, etc., but is not limited to the above forms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0095] Figure 3 is an optional flowchart of a method for mining strongly connected components of graph data provided in an embodiment of the present application. Figure 3 The method includes steps 110 to 130.

[0096] Step 110, obtaining a directed graph; the directed graph includes a plurality of nodes, each node corresponds to a first label, the first label includes a first in-degree attribute value and a first out-degree attribute value; different nodes correspond to different first in-degree attribute values, and different nodes correspond to different first out-degree attribute values;

[0097] In the embodiments of the present application, there is no restriction on the source of the directed graph obtained. For example, the directed graph can be graph data generated based on relevant information in the application scenario, or it can be graph data remaining after other mining algorithms have performed some strongly connected component mining tasks; in other words, the method in the embodiments of the present application can perform mining tasks starting from complete graph data alone, or it can be used to process mining tasks that are not completed by other algorithms.

[0098] Reference Figure 4 , Figure 4 , a schematic diagram of the structure of a directed graph obtained is shown in FIG. 1 , in which there are multiple nodes, and edges representing connection paths exist between some nodes. It can be understood that Figure 4 It is only an example of the structure of a directed graph and does not mean to limit the structure of the graph data processed in the actual application of the method. In other words, the mining method in the embodiment of the present application is applicable to graph data including various numbers of nodes and various connection relationships between nodes.

[0099] For the convenience of description, Figure 4 Each node is numbered to distinguish it, and different nodes have different numbers. Specifically, in this directed graph, each circle represents a node, and the number in the circle represents the number corresponding to the node. Figure 4 The nodes in the directed graph are numbered from 0 to 10, and the nodes can be recorded as the zeroth node 0, the first node 1, the second node 2, and the tenth node 10. In a directed graph, the edge between two nodes is directed. For example, the edge between the first node 1 and the second node 2 points from the second node 2 to the first node 1, indicating that there is a directed path from the second node 2 to the first node 1 between the two nodes. Of course, there may be a bidirectional connection relationship between some nodes, such as Figure 4 In the ninth node 9 and the tenth node 10, there is an edge from the ninth node 9 to the tenth node 10, and there is an edge from the tenth node 10 to the ninth node 9. In this case, the ninth node 9 and the tenth node 10 can be called a node pair, and the nodes in each node pair can be recorded as the second child node and the third child node. For example Figure 4 The ninth node 9 in can be recorded as the second child node of the node pair, and correspondingly, the tenth node 10 can be recorded as the third child node of the node pair.

[0100] For each node and edge in a directed graph, the specific meaning represented by it can be flexibly set according to the application field of the graph data: for example, in some embodiments, a directed graph can represent a group of user groups, then each node therein can represent a user, and the edge between nodes can represent the attention relationship between two users. For example, the first node 1 can represent user A, the second node 2 can represent user B, and the directed path from the second node 2 to the first node 1 represents that user B follows user A. In some embodiments, a directed graph can represent a group of communication networks, then each node therein can represent a communication device, and the edge between nodes can represent the information transmission relationship between two communication devices. For example, the first node 1 can represent communication device C, the second node 2 can represent communication device D, and the directed path from the second node 2 to the first node 1 represents that communication device D can send information to communication device C.

[0101] In the embodiment of the present application, each node also corresponds to a label, which is used for subsequent label propagation to determine the connectivity of each node and other nodes. These labels are recorded as first labels, and each first label includes two values, which are recorded as the first in-degree attribute value and the first out-degree attribute value respectively. The first label can be expressed as "(first in-degree attribute value, first out-degree attribute value)". In the initial state, the values ​​in the first labels corresponding to different nodes are all different, that is, the first in-degree attribute values ​​corresponding to different nodes are different, and the first out-degree attribute values ​​corresponding to different nodes are also different. For the convenience of description, when the embodiment of the present application is initialized, the number corresponding to each node can be used as the first in-degree attribute value and the first out-degree attribute value corresponding to the node. For example, for the first node 1, its corresponding number is 1, then the value 1 can be used as the first in-degree attribute value and the first out-degree attribute value of the first node 1, so the label corresponding to the first node can be expressed as (1, 1). Similarly, the initial first labels of other nodes can be set in the above manner, so that the conditions of different first in-degree attribute values ​​and first out-degree attribute values ​​corresponding to each node can be achieved more easily. Of course, it should be noted here that the above initialization of the first label of each node is a possible implementation method. In fact, when setting the first label, it is sufficient to satisfy that the first in-degree attribute values ​​corresponding to different nodes are different, and the first out-degree attribute values ​​corresponding to different nodes are different. In addition, in some embodiments, the mining task processed by this method may be the graph data remaining after other algorithms have processed part of it. If these graph data also have the first label, and the initialization conditions of the first label in other algorithms are the same as those of this method, then this method does not need to re-initialize the first label of each node when processing these graph data, so as to save computing resources and improve mining efficiency.

[0102] Step 120, dividing the directed graph into a plurality of block graphs;

[0103] In the embodiment of the present application, after obtaining the directed graph to be processed, the directed graph can be divided into multiple block graphs. Specifically, the block graph here can be any combination of nodes and edges in the directed graph. There are many ways to divide the directed graph: in some embodiments, the directed graph can be randomly divided into several parts according to the graph structure, for example, referring to Figure 5 , the directed graph 510 is randomly divided into two parts, namely block graph 521 and block graph 522, wherein the block graph 521 includes 8 nodes including the zeroth node 0, the first node 1, and the edges between these nodes; the block graph 522 includes 3 nodes including the eighth node 8, the ninth node 9 and the tenth node 10, and the edges between these nodes. Figure 5 The example in the example just separates two groups of nodes that are not connected in the directed graph. In fact, other situations may also occur when dividing the block graph: Figure 6 ,exist Figure 6 In the example, the directed graph 510 is randomly divided into two parts, namely, the block graph 531 and the block graph 532. The block graph 531 includes 6 nodes from the 0th node 0 to the 5th node 5, and the edges between these nodes; the block graph 532 includes 5 nodes from the 6th node 6 to the 10th node 10, and the edges between these nodes. It should be noted that in Figure 6 In the process of dividing the directed graph 510 into the block graph 531 and the block graph 532, the edges between some nodes are cut apart, such as the edge of the fifth node 5 pointing to the seventh node 7. In order to facilitate the update of the attribute value in the subsequent label propagation process, in the embodiment of the present application, while dividing the block graph, the connection relationship of each node and the attribute value of the adjacent node can be recorded and stored as the corresponding connection node information at each node. For example, for the fifth node 5, the nodes that have a direct connection relationship with it include the third node 3, the fourth node 4 and the seventh node 7, wherein the nodes connected along the out-degree direction include the third node 3, the fourth node 4 and the seventh node 7, and the nodes connected along the in-degree direction include the third node 3. At this time, the connection nodes of the fifth node 5 and the corresponding in-degree attribute values ​​or out-degree attribute values ​​of these nodes can be recorded according to the in-degree and out-degree respectively. Specifically, the connection nodes in the out-degree direction of the fifth node 5 can be recorded as {third node 3, fourth node 4, seventh node 7}, and the corresponding out-degree attribute values ​​can be recorded as {3, 4, 7}; the connection nodes in the in-degree direction of the fifth node 5 can be recorded as {third node 3}, and the corresponding in-degree attribute value (i.e., the in-degree attribute value of the third node 3) is {3}.

[0104] Based on the above understanding, in some embodiments, the directed graph can also be randomly divided into several parts according to the nodes. Since each node can save its corresponding connection node information, it is not necessary to stick to the actual connection relationship between the nodes to divide the directed graph. For example, in a possible implementation of the division method, the directed graph can be divided into a first block diagram (including the first node 1, the fourth node 4 and the eighth node 8), a second block diagram (including the second node 2, the third node 3 and the seventh node 7) and a third block diagram (including the fifth node 5, the sixth node 6, the ninth node 9 and the tenth node 10). Although there may be no actual connection relationship between the nodes, based on the connection node information corresponding to each node in the block diagram, label propagation updates can still be performed normally.

[0105] Step 130: Asynchronously update the first in-degree attribute value and the first out-degree attribute value of the nodes in each block graph through label propagation, and after all nodes in the directed graph are updated, determine that the nodes carrying the same first label as the first child node belong to the same strongly connected component; the first child node is the node with the same first label before and after the update.

[0106] In the embodiment of the present application, for the nodes in each block diagram, an asynchronous label propagation update can be performed. Label propagation update refers to updating the first label of a node according to the first labels of other nodes that can be reached by a node along the out-degree direction or the first labels of other nodes that can be reached by the node along the in-degree direction. For example, Figure 5 For example, for the first node 1 in the block diagram 521, its initial first label is (1, 1), and the in-degree direction can reach other nodes of the first node 1, that is, the nodes that can reach the first node 1 along the directed edge include the zeroth node 0 (reached by the second node 2) and the second node 2, so the in-degree attribute value of the first node 1 can be updated according to the first in-degree attribute values ​​of the zeroth node 0 and the second node 2. Generally speaking, when updating the first in-degree attribute value of the first node 1, in order to facilitate distinction and calculation, the maximum or minimum first in-degree attribute value can be selected from the first in-degree attribute values ​​of the zeroth node 0 and the second node 2 as the updated first in-degree attribute value of the first node 1. Since the maximum (or minimum) attribute value has convergence, if a node can be reached by the node with the maximum (or minimum) in-degree attribute value among all the nodes in the graph data, then the first in-degree attribute value of the node will definitely be updated to the maximum (or minimum) in-degree attribute value. In the embodiment of the present application, for the sake of convenience of description, the update to the minimum attribute value is taken as an example. At this time, for the first node 1, among the nodes that can reach it, the node with the smallest first in-degree attribute value is the zeroth node 0, and the smallest first in-degree attribute value is 0. Therefore, the first in-degree attribute value of the first node 1 can be updated to 0 through the label propagation algorithm.

[0107] The above process illustrates the principle of label propagation update. In the actual update process, the first in-degree attribute value of the first node 1 is not directly updated to 0, but needs to be updated to 0 after several rounds of iterations. Figure 7 , Figure 7 The specific updating process of the first in-degree attribute value of each node in the block diagram 521 is shown. Figure 7In the first round of update, the specific implementation process of the label propagation algorithm is described. When updating the first in-degree attribute value of each node, the in-degree connection node of each node is first determined: for example, for the first node 1, its in-degree connection node is the second node 2, and for the fourth node 4, its in-degree connection nodes include the second node 2, the fifth node 5 and the sixth node 6. Then the first in-degree attribute value of the in-degree connection node corresponding to each node is determined, and the attribute value is recorded as the third in-degree attribute value corresponding to the node: for example, the third in-degree attribute value corresponding to the first node 1 is 2; the fourth in-degree attribute value corresponding to the fourth node 4 is 2, 5 and 6, and then the minimum value of the original first in-degree attribute value and the third in-degree attribute value of each node is used as the first in-degree attribute value after the node is updated. For example, the original first in-degree attribute value corresponding to the first node 1 is the smallest, so the first in-degree attribute value of the first node 1 after this round of update is still 1; and the value 2 of the third in-degree attribute value corresponding to the fourth node 4 is the smallest, so the first in-degree attribute value of the fourth node 4 after this round of update is 2. It can be understood that in the embodiment of the present application, the maximum value of the first in-degree attribute value and the first out-degree attribute value corresponding to the node can also be used as the update target. The implementation principle is similar to the minimum value and will not be elaborated here.

[0108] Figure 7 For block graph 521, when the first labels of all nodes are updated once (the update can also keep the original value), it is considered that one round of label propagation update is completed. Repeat the above update process, and when the first in-degree attribute values ​​of all nodes in block graph 521 no longer change, it can be considered that the label propagation update of the in-degree direction of block graph 521 is completed. Figure 7 It can be seen that after four rounds of iterative updates, the first in-degree attribute values ​​of all nodes in the graph are updated to 0, indicating that this part of the block graph has completed the label propagation update in the in-degree direction.

[0109] Of course, in the embodiment of the present application, the label propagation update of the first label also includes the update of the first out-degree attribute value. The update process of the first out-degree attribute value is similar to the update process of the first in-degree attribute value. The first out-degree attribute value of the out-degree connection node of each node is also collected and recorded as the third out-degree attribute value. Then, the minimum or maximum value between the original first out-degree attribute value of each node and the corresponding third out-degree attribute value is used as the first out-degree attribute value of the node after the update. It will not be elaborated here.

[0110] In the embodiment of the present application, when the first labels of all nodes in the directed graph no longer change, it is considered that all nodes in the directed graph have been updated. Figure 8 , taking block graph 521 as a separate directed graph, and updating its label propagation without considering the connection relationship between it and other block graphs, Figure 8 As shown in the diagram after completing the label propagation update, it can be seen that the first label corresponding to the first node 1 of block diagram 521 is updated from (1, 1) to (0, 0). For the directed graph after the first label is updated, the starting node that can reach more nodes and is reached by more nodes can be determined, that is, the node with the same first label before and after the update, which is recorded as the first child node. Since the first child node is not affected by the first labels of other nodes, it can be determined that it may have updated the first labels of the batch nodes (such as Figure 8 The zeroth node 0 in the graph is found, so we can start from the first child node and perform a depth-first search along the in-degree or out-degree direction to determine all the searched nodes that carry the same first label as the first child node (including the first child node itself). The first labels of these nodes except the first child node are the same as the first child node, indicating that they can all reach the first child node along a directed path and can also be reached by the first child node along a directed path. Therefore, we can determine that these nodes can reach each other in pairs, and these nodes and the first child node can constitute a strongly connected subgraph of the directed graph. Figure 8 In , the zeroth node 0 is the first child node, and the nodes carrying the same first label (0, 0) as the zeroth node 0 include the first node 1 and the second node 2. The zeroth node 0, the first node 1 and the second node 2 belong to the same strongly connected subgraph. On the other hand, in addition to the above nodes, the other nodes have different first labels from the first child node, which means that they cannot reach the first child node along a directed path or be reached by the first child node along a directed path. Therefore, it means that the other nodes and the first child node are not strongly connected, that is, the other nodes and the first child node do not belong to a strongly connected component, so the strongly connected subgraph constructed above is a maximal strongly connected subgraph of the directed graph, that is, it belongs to the same strongly connected component. For example Figure 8 In the graph, except for the first node 1 and the second node 2, all other nodes cannot reach the zeroth node 0 along the directed path. It can be judged that other nodes cannot reach any of the zeroth node 0, the first node 1 and the second node 2. Therefore, the graph data composed of the zeroth node 0, the first node 1 and the second node 2 is a strongly connected component.

[0111] The above describes the process of updating the label propagation of nodes in each block diagram of the embodiment of the present application, and different block diagrams are asynchronous when updating the labels of nodes. The following describes the asynchronous propagation update principle of the first label in the present application.

[0112] Reference Fig. 9 and Fig.10 , Fig. 9 and Fig.10Schematic diagrams of some optional implementation environments for asynchronous label propagation updates in embodiments of the present application are shown. Fig. 9 In the implementation environment shown, multiple processing units can be set up when asynchronous label propagation updates are performed on a directed graph. Each processing unit can be a terminal device. A central processing unit is selected to store the relevant data of the directed graph, and other processing units communicate with the central processing unit in a two-way manner. Then, after the directed graph is divided into multiple block graphs, each block graph and the connection node information corresponding to the nodes in the block graph are sent to other processing units. The first label of a node in a block graph is processed in one processing unit, and one processing unit can process multiple block graphs in sequence. Fig. 9 In the embodiment, terminal device 911, terminal device 912, terminal device 913, terminal device 914, and terminal device 915 can be configured as processing units for processing label propagation updates of nodes in the block graph; terminal device 921 can be configured as a central processing unit for storing relevant data of the directed graph.

[0113] exist Fig.10 In the illustrated implementation environment, the related data of the directed graph can be stored in the cloud server 1020, and then after the directed graph is divided into a plurality of block graphs, the connection node information corresponding to each block graph and the nodes in the block graph is sent to an independent processing unit, such as Fig.10 The terminal device 1011, the terminal device 1012 and the terminal device 1013 in the block diagram can all be used as a processing unit; similarly, the first label of a node in a block diagram is processed in a processing unit, and a processing unit can process multiple block diagrams in sequence. Specifically, the above process can be implemented on the Spark on Angel high-performance graph computing platform, which integrates a parameter server and multiple distributed processing units, cancels the original synchronization of pulling and uploading on the platform, and uses only one data structure to transmit information. Whether the processing unit uploads information to the server or pulls information, it is only carried out through this data structure.

[0114] Combine the following Fig.11 The principle and advantages of asynchronous updating in the embodiment of the present application are described. Fig.11 The data transmission principle when the first label is asynchronously updated in the embodiment of the present application is shown. In the embodiment of the present application, the asynchronous update process can be simply summarized as follows: the first label of each node in the directed graph is stored, the first label of the node in each block graph is asynchronously propagated and updated according to the stored first label, and the stored first label is updated according to the result of the label propagation update, and the above process is iterated repeatedly until the first labels of all nodes in the directed graph no longer change, and it is considered that all nodes are updated.

[0115] Taking the update of the first in-degree attribute value as an example, the first in-degree attribute value of each node in the directed graph is first stored in the server, and then in each processing unit, the first in-degree attribute value and the third in-degree attribute value corresponding to one or more nodes in the block graph are asynchronously pulled, and the first in-degree attribute value of the node in the block graph is updated according to this information, and the updated first in-degree attribute value is synchronized to the server. Fig.11 In the example, two processing units asynchronously update the first label, which are respectively recorded as the first processing unit 1110 and the second processing unit 1120. The first processing unit 1110 pulls a batch of block diagram data S1 at a certain moment, and sequentially processes and uploads two groups of block diagram data S2 and data S4 after updating the first label. When the second processing unit 1120 pulls the block diagram data S3 to be processed, the connection node information corresponding to some nodes may be updated by the data S2. For example, refer to Figure 6 , assuming that the block graph 531 is pulled by the first processing unit 1110 in the data S1, and the first label is updated in the data S2 and uploaded to the server, and the second processing unit 1120 happens to process the block graph 532. When it pulls the data S3 containing the block graph 532, the connection node information of some nodes in the data S3 (such as the sixth node 6 and the seventh node 7) has been updated, then the second processing unit 1120 can pull more useful data in advance. And because the direction of comparison and convergence in the first label update process is consistent, that is, in each processing unit, the smaller (or larger) attribute value will always be saved, so the information of label propagation is asynchronously compatible. The advantage of asynchronous update is that the first label update of the directed graph as a whole is divided into batches, and each batch performs a partial update of the first label data on the server side. Then, when executing the pull operation on the parallel processing unit, it is possible to pull the most recently updated attribute value, which is equivalent to pulling the value of the next round of the label propagation process in advance. Through this update method, the number of iterations can be greatly reduced. In theory, the more asynchronous pulls occur, the more likely it is that the number of iteration rounds will be significantly reduced.

[0116] In the aforementioned embodiment, the first label in the directed graph is updated by a parallel asynchronous label propagation update algorithm, which is conducive to reducing the amount of redundant calculations and reducing the number of iterations of the entire mining algorithm. However, since the distributed mining method will always have the problem of redundant calculations more or less, in the embodiment of the present application, after mining the strongly connected component, the strongly connected component can be extracted from the directed graph to reduce the amount of data of the entire directed graph. In addition, when it is determined that the amount of data of the directed graph can be executed on a single machine, it is promptly transferred to the mining algorithm executed on a single machine to complete the mining task more efficiently. Specifically, when mining on a single machine, it is equivalent to sending the remaining directed graph data to a single processing unit for mining. The storage capacity of the processing unit should be able to accommodate the total amount of data of the remaining directed graph. In some embodiments, the number of nodes in the directed graph can be detected in real time. When the number of nodes is less than or equal to the preset first threshold, it is considered that the demand for single-machine execution can be met. In some embodiments, the number of connection relationships between nodes in the directed graph, that is, the number of directed edges, can also be detected in real time. When the number of connection relationships is less than or equal to the preset second threshold, it is considered that the data volume limit for single-machine execution can be met. Here, the actual size of the first threshold and the second threshold can be determined and adjusted according to the data storage capacity of a single processing unit, and this is not limited in the present application. In some embodiments, after the determined strongly connected components are extracted from the directed graph to obtain an updated directed graph, if the data size of the updated directed graph is still large, for example, the number of nodes may be greater than the first threshold or the number of connection relationships may be greater than the second threshold, it is considered that the remaining directed graph cannot be mined on a single machine. At this time, the first labels of the nodes in the remaining directed graph can be initialized, and then the process of dividing the block graph in step 120 is returned to re-execute the mining method in the embodiment of the present application, and continue to extract the mined strongly connected components therefrom until the graph data size of the remaining directed graph can be executed on a single machine.

[0117] Below, refer to Fig.12 The process of performing strongly connected component mining on a single machine in the embodiment of the present application is described by using the directed graph in the example: in a single processing unit, after obtaining a directed graph that can be mined on a single machine, a depth-first search is first performed on the directed graph, and the timestamp when each node is searched is determined, recorded as the first timestamp, and the minimum timestamp that each node can be traced back to, recorded as the second timestamp. For example Fig.12The directed graph in includes six nodes, namely the first node 1 to the sixth node 6. The order in which each node is searched is the first node 1, the second node 2, the third node 3, the sixth node 6, the fourth node 4 and the fifth node 5. The first timestamp of each node being searched can be represented by DFN[]. For example, DFN[1]=1 means that the first node is searched first. Similarly, DFN[6]=4 means that the sixth node is searched fourth. The timestamp that the node can be traced back to is the earliest node that can be reached in the out-degree direction of each node. The second timestamp of the node can be represented by LOW[]. For example, LOW[5]=1 means that the fifth node 5 can reach the first node 1 that is searched first, and the second timestamp corresponding to the fifth node 5 is 1.

[0118] Then, each node is put into a stack in the order in which it is searched, and a search is started from the top of the stack for a node with the same first timestamp and second timestamp, and this node is recorded as the fourth child node, for example Fig.12 In the example, the two timestamps of the fifth node 5 at the top of the stack are 6 and 1, which are different. Continue searching in the stack; the two timestamps of the fourth node are 5 and 1, which are also different. Continue searching in the stack; the two timestamps of the sixth node 6 are the same, both 4, and it is determined that the timestamps of other nodes are not the same as 4, then the sixth node 6 can be popped out of the stack as a strongly connected component, and the search can be continued to find that the third node 3 is a separate strongly connected component, and the first node 1, the second node 2, the fourth node 4 and the fifth node 5 have the same second timestamp, which can be considered to belong to the same strongly connected component. That is, Fig.12 The directed graph shown can mine three strongly connected components, which are respectively recorded as {sixth node 6}, {third node 3} and {first node 1, second node 2, fourth node 4 and fifth node 5}. In the embodiment of the present application, when the single-machine mining is completed, the aforementioned extracted strongly connected components and the strongly connected components obtained by the single-machine mining can be sorted to obtain the strongly connected component mining result of the input directed graph.

[0119] In some embodiments, before performing strongly connected component mining, the directed graph may be subjected to bidirectional edge compression and bilateral decoupling processing to reduce the data volume of the entire directed graph. Specifically, when performing bidirectional edge compression, refer to Fig.13 ,by Figure 5For example, the directed graph 510 in the directed graph 510 can first be determined. The node pair is the second child node and the third child node of the aforementioned bidirectional connection relationship. For example, the third node 3 and the fifth node 5 in the directed graph 510 belong to a node pair, the sixth node 6 and the fourth node 4 also belong to a node pair, and the ninth node 9 and the tenth node 10 also belong to a node pair. After determining the node pair, because the nodes in the node pair can reach each other, the two nodes are strongly connected and can be compressed and merged into one node. In the embodiment of the present application, when compressing the node pair, any one of the nodes can be retained separately as the compressed node pair. For example, the node pair of the sixth node 6 and the fourth node 4 can be compressed into the fourth node 4, and of course, it can also be compressed into the sixth node 6, and the compression method of each other node pair is similar. In addition, it should be noted that in some embodiments, after a round of bidirectional edge compression is performed, new bidirectional edges may be generated in the new directed graph. This is because the compressed node pair may form a new node pair with other nodes. For example, after the node pair including the ninth node 9 and the tenth node 10 in the directed graph 510 is compressed into one node (taking compression into the ninth node 9 as an example), it will have a bidirectional connection relationship with the eighth node 8. Therefore, the compression can continue at this time, so that the eighth node 8, the ninth node 9 and the tenth node 10 can be compressed into one node. It can be understood that through the above bidirectional edge compression processing, multiple nodes in the original directed graph can be compressed into one node. For a complex directed graph, after performing multiple bidirectional edge compression operations, the scale of the graph will be significantly reduced, which can reduce the amount of data processing.

[0120] When bilateral decoupling is performed, the labels of each node in the directed graph can be propagated and updated first, and the updated labels are recorded as the second labels, and the attribute values ​​in the second labels are recorded as the second in-degree attribute values ​​and the second out-degree attribute values. The label propagation update algorithm here can be executed by a conventional algorithm or by the aforementioned asynchronous label propagation update method. Figure 8 For example, Figure 8 The updated label in is the second label. Fig.14 In the embodiment of the present application, the decoupled edge in the directed graph is determined according to the second label, and the decoupled edge is an edge whose nodes at both ends of the directed edge correspond to different second labels, for example Fig.14 In the example, the second label of the fifth node 5 is (0, 3), and the second label of the fourth node 4 is (0, 4). The edge between the fifth node 5 and the fourth node 4 is a decoupled edge. Since the second labels of the nodes at both ends of the decoupled edge are not completely consistent after the label propagation update, it means that the two nodes are in different strongly connected components. Therefore, the edge between the two nodes can be deleted, thereby reducing the coupling degree between the two strongly connected components and facilitating the subsequent mining of strongly connected components.

[0121] It should be noted that in the embodiment of the present application, when performing the above-mentioned bidirectional edge compression and bilateral decoupling processing operations, the data size of the directed graph can also be detected in real time. When the requirements for single-machine execution are met, it is promptly transferred to the processing unit for processing to improve the efficiency of mining.

[0122] The following describes an implementation process of the method of the present application in conjunction with a specific embodiment.

[0123] Reference Fig.15 In the embodiment of the present application, after obtaining the graph data to be processed, the first label of each node in the graph data is initialized, and then Fig.13 The method shown is to perform bidirectional edge compression on the directed graph, compress and merge some of the nodes, and determine whether the data volume of the compressed graph data can be mined on a single machine. If the data volume of the remaining graph data is small, it can be mined on a single machine, and then the graph data is sent to the single machine for mining. If the data volume of the remaining graph data is large and cannot be mined on a single machine, then it can be mined on a single machine. Fig.14 The remaining graph data is bidirectionally decoupled in the manner shown, and some directed edges are deleted. For the decoupled graph data, continue to determine whether the data volume can be mined on a single machine. If it can, send the graph data to a single machine for mining. If it still cannot be satisfied, then follow the steps in this application. Figure 3 The principle of the flowchart shown is to execute the strongly connected component mining method of graph data provided in the embodiment of the present application to perform distributed asynchronous mining on directed graphs.

[0124] Moreover, during the asynchronous mining process, after each round of strongly connected component mining is completed, it is continuously determined whether the data volume can be mined on a single machine. If the single machine execution requirement is not met, the first label of the remaining graph data is initialized and the distributed asynchronous mining is performed again. The above asynchronous mining work is repeated until the remaining graph data can be executed on a single machine, and the remaining graph data is sent to a single processing unit for mining. Finally, when the graph data is executed on a single machine after the mining algorithm is executed, the strongly connected components obtained in the two processes of single machine mining and distributed asynchronous mining are merged and sorted to obtain the mining results of all strongly connected components of the graph data.

[0125] It should be noted that Fig.15 The implementation process shown is an optional implementation method. In actual execution, the specific process can be adjusted as needed. For example, in some embodiments, the order of bidirectional edge compression processing and bidirectional decoupling processing can be changed. Of course, some of the processing processes can also be removed. This is not limited in the embodiments of the present application.

[0126] Reference Fig.16 The present application also discloses a device for mining strongly connected components of graph data, including:

[0127] The acquisition module 1610 is used to acquire a directed graph; the directed graph includes a plurality of nodes, each node corresponds to a first label, the first label includes a first in-degree attribute value and a first out-degree attribute value; different nodes correspond to different first in-degree attribute values, and different nodes correspond to different first out-degree attribute values;

[0128] A partitioning module 1620, configured to partition the directed graph into a plurality of block graphs;

[0129] The storage and update module 1630 is used to store the first label of each node in the directed graph, perform asynchronous label propagation update on the first label of each node in the block graph according to the stored first label, and update the stored first label according to the result of the label propagation update;

[0130] Processing module 1640 is used to determine that nodes carrying the same first label as the first child node belong to the same strongly connected component after all nodes in the directed graph are updated; the first child node is a node with the same first label before and after the update.

[0131] Optionally, in some embodiments, the device further comprises:

[0132] The bidirectional edge compression module is used to determine the node pairs in the directed graph and merge the node pairs into one node; the node pair includes a second child node and a third child node, and the second child node and the third child node have a bidirectional connection relationship.

[0133] Optionally, in some embodiments, the device further comprises:

[0134] A propagation module, used for performing label propagation update on the first in-degree attribute value and the first out-degree attribute value of the node in the directed graph to obtain a second label of each node; the second label includes the second in-degree attribute value and the second out-degree attribute value;

[0135] The decoupling module is used to determine and delete the decoupling edge in the directed graph according to the second label; the second labels corresponding to the nodes connected at both ends of the decoupling edge are different.

[0136] Optionally, in some embodiments, the storage and update module includes:

[0137] A first storage module, used for storing the first in-degree attribute value of each node in the directed graph to the server;

[0138] A first pulling module is used to asynchronously pull the first in-degree attribute value of the node in each block graph and the third in-degree attribute value corresponding to each node through multiple processing units; wherein each processing unit processes at least one block graph; the third in-degree attribute value is the first in-degree attribute value of the in-degree connection node of the node;

[0139] The first updating module is used to update the first in-degree attribute value of the node in the processing unit according to the third in-degree attribute value corresponding to the node, and synchronize the updated first in-degree attribute value to the server.

[0140] Optionally, in some embodiments, the first update module includes:

[0141] A first determination submodule, used to determine the minimum or maximum value of the first in-degree attribute value and the third in-degree attribute value corresponding to the node;

[0142] The first processing submodule is used to use the minimum value or the maximum value as the first in-degree attribute value after the node is updated.

[0143] Optionally, in some embodiments, the storage and update module includes:

[0144] A second storage module is used to store the first out-degree attribute value of each node in the directed graph to the server;

[0145] The second pulling module is used to asynchronously pull the first out-degree attribute value of the node in each block graph and the third out-degree attribute value corresponding to each node through multiple processing units; wherein each processing unit processes at least one block graph; the third out-degree attribute value is the first out-degree attribute value of the out-degree connection node of the node;

[0146] The second updating module is used to update the first out-degree attribute value of the node in the processing unit by the third out-degree attribute value corresponding to the node, and synchronize the updated first out-degree attribute value to the server.

[0147] Optionally, in some embodiments, the second update module includes:

[0148] A second determination submodule is used to determine the minimum value or the maximum value of the first out-degree attribute value and the third out-degree attribute value corresponding to the node;

[0149] The second processing submodule uses the minimum value or the maximum value as the first out-degree attribute value after the node is updated.

[0150] Optionally, in some embodiments, the device further comprises:

[0151] The extraction module is used to extract the determined strongly connected components from the directed graph to obtain an updated directed graph.

[0152] Optionally, in some embodiments, the device further comprises:

[0153] The first detection submodule is used to detect the number of nodes in the directed graph;

[0154] A third processing submodule is configured to send the directed graph to a single processing unit for strongly connected component mining when the number of nodes is less than or equal to the first threshold; or

[0155] The second detection submodule is used to detect the number of connection relationships between nodes in the directed graph;

[0156] The fourth processing submodule is used to send the directed graph to a single processing unit for strongly connected component mining when the number of connection relationships is less than or equal to the second threshold.

[0157] Optionally, in some embodiments, the device further comprises:

[0158] The initialization module is used to initialize the first label corresponding to each node in the updated directed graph when the number of nodes is greater than the first threshold or when the number of connection relationships is greater than the second threshold, and return to the division module to execute the step of dividing the directed graph into multiple block graphs.

[0159] Optionally, in some embodiments, the third processing submodule or the fourth processing submodule includes:

[0160] A sending submodule, used for sending the directed graph to a single processing unit;

[0161] A first search submodule is used to perform a depth-first search on the directed graph in the processing unit, and add each node to the stack in the order in which it is searched;

[0162] A second search submodule is used to search for a fourth subnode from the top of the stack to the middle of the stack in sequence; the first timestamp and the second timestamp corresponding to the fourth subnode are the same; wherein the first timestamp is the timestamp when each node is searched, and the second timestamp is the earliest timestamp that each node can be traced back to;

[0163] The mining submodule is used to determine that the nodes having the same first timestamp or second timestamp as the fourth subnode belong to the same strongly connected component.

[0164] Optionally, in some embodiments, the processing module includes:

[0165] A fifth processing submodule, configured to determine a first child node from the directed graph after all nodes in the directed graph are updated;

[0166] The third search submodule is used to start from the first child node and perform a depth-first search along the direction of the in-degree or out-degree to determine that the nodes carrying the same first label as the first child node belong to the same strongly connected component.

[0167] Understandably, Figure 3 The contents of the graph data strongly connected component mining method embodiment shown in the figure are all applicable to the present device embodiment, and the functions specifically implemented by the present device embodiment are the same as those in the present device embodiment. Figure 3 The method for mining strongly connected components of graph data shown in the embodiment is the same as that of Figure 3 The beneficial effects achieved by the embodiment of the method for mining strongly connected components of graph data shown are also the same.

[0168] Reference Fig.17 , the embodiment of the present application further discloses a computer device, including:

[0169] at least one processor 1710;

[0170] At least one memory 1720, used to store at least one program;

[0171] When at least one program is executed by at least one processor 1710, at least one processor 1710 implements the following Figure 3 An embodiment of a method for mining strongly connected components of graph data is shown.

[0172] Understandably, Figure 3 The contents of the graph data strongly connected component mining method embodiment shown in the figure are all applicable to the computer device embodiment, and the functions specifically implemented by the computer device embodiment are the same as those in the embodiment of FIG. Figure 3 The method for mining strongly connected components of graph data shown in the embodiment is the same as that of Figure 3 The beneficial effects achieved by the embodiment of the method for mining strongly connected components of graph data shown are also the same.

[0173] The present application also discloses a computer-readable storage medium in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the following Figure 3 An embodiment of a method for mining strongly connected components of graph data is shown.

[0174] Understandably, Figure 3 The contents of the embodiment of the method for mining strongly connected components of graph data shown in the figure are all applicable to the embodiment of this computer-readable storage medium. The functions specifically implemented by this computer-readable storage medium embodiment are similar to those of Figure 3 The method for mining strongly connected components of graph data shown in the embodiment is the same as that of Figure 3 The beneficial effects achieved by the embodiment of the method for mining strongly connected components of graph data shown are also the same.

[0175] The embodiment of the present application further discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in the above-mentioned computer-readable storage medium; Fig.17 The processor of the computer device shown can read the computer instruction from the above-mentioned computer readable storage medium, and the processor executes the computer instruction so that the computer device executes Figure 3 The method for mining strongly connected components of graph data is shown.

[0176] Understandably, Figure 3 The contents of the embodiments of the method for mining strongly connected components of graph data shown in the figure are all applicable to the computer program product or computer program embodiment. The functions specifically implemented by the computer program product or computer program embodiment are similar to those of the embodiment of the method for mining strongly connected components of graph data shown in the figure. Figure 3 The method for mining strongly connected components of graph data shown in the embodiment is the same as that of Figure 3 The beneficial effects achieved by the embodiment of the method for mining strongly connected components of graph data shown are also the same.

[0177] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0178] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0179] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0180] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0181] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0182] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0183] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0184] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0185] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for mining strongly connected components in graph data. It is characterized in that The following steps are involved: Obtain a directed graph; the directed graph includes a plurality of nodes, each of the nodes corresponds to a first label, the first label includes a first in-degree attribute value and a first out-degree attribute value; different nodes correspond to different first in-degree attribute values, and different nodes correspond to different first out-degree attribute values; Dividing the directed graph into a plurality of block graphs; The first label of each node in the directed graph is stored, the first label of each node in the block graph is asynchronously updated by label propagation according to the stored first label, and the stored first label is updated according to the result of the label propagation update; When all nodes in the directed graph are updated, it is determined that the nodes carrying the same first label as the first child node belong to the same strongly connected component; the first child node is a node having the same first label before and after the update; The storing of the first label of each node in the directed graph, performing asynchronous label propagation update on the first label of each node in the block graph according to the stored first label, and updating the stored first label according to the result of the label propagation update, comprises: storing the first in-degree attribute value of each of the nodes in the directed graph to a server; Asynchronously pulling the first in-degree attribute value of the node in each of the block graphs and the third in-degree attribute value corresponding to each of the nodes through multiple processing units; wherein each of the processing units processes at least one of the block graphs; the third in-degree attribute value is the first in-degree attribute value of the in-degree connection node of the node; In the processing unit, the first in-degree attribute value of the node is updated by the third in-degree attribute value corresponding to the node, and the updated first in-degree attribute value is synchronized to the server.

2. The method according to claim 1, It is characterized in that The method further comprises the following steps: A node pair in the directed graph is determined, and the node pair is merged into one node; the node pair includes a second child node and a third child node, and a bidirectional connection relationship exists between the second child node and the third child node.

3. The method according to claim 1, It is characterized in that The method further comprises the following steps: Perform label propagation update on the first in-degree attribute value and the first out-degree attribute value of the node in the directed graph to obtain a second label of each node; the second label includes a second in-degree attribute value and a second out-degree attribute value; Determine and delete a decoupling edge in the directed graph according to the second label; the nodes connected at both ends of the decoupling edge have different second labels.

4. The method according to claim 1, It is characterized in that The updating the first in-degree attribute value of the node by using the third in-degree attribute value corresponding to the node includes: Determine a minimum value or a maximum value of the first in-degree attribute value and the third in-degree attribute value corresponding to the node; The minimum value or the maximum value is used as the first in-degree attribute value after the node is updated.

5. The method according to claim 1, It is characterized in that The storing of the first label of each node in the directed graph, performing asynchronous label propagation update on the first label of each node in the block graph according to the stored first label, and updating the stored first label according to the result of the label propagation update, comprises: Storing the first out-degree attribute value of each of the nodes in the directed graph to a server; Asynchronously pulling the first out-degree attribute value of the nodes in each of the block graphs and the third out-degree attribute value corresponding to each of the nodes through multiple processing units; wherein each of the processing units processes at least one of the block graphs; the third out-degree attribute value is the first out-degree attribute value of the out-degree connection node of the node; In the processing unit, the first out-degree attribute value of the node is updated by the third out-degree attribute value corresponding to the node, and the updated first out-degree attribute value is synchronized to the server.

6. The method according to claim 5, It is characterized in that The updating the first out-degree attribute value of the node by using the third out-degree attribute value corresponding to the node includes: Determine a minimum value or a maximum value of the first out-degree attribute value and the third out-degree attribute value corresponding to the node; The minimum value or the maximum value is used as the first out-degree attribute value after the node is updated.

7. The method according to any one of claims 1 to 6, It is characterized in that The method further comprises the following steps: The determined strongly connected components are extracted from the directed graph to obtain an updated directed graph.

8. The method according to claim 7, It is characterized in that The method further comprises the following steps: Detecting the number of nodes in the directed graph; When the number of nodes is less than or equal to a first threshold, sending the directed graph to a single processing unit for strongly connected component mining; or, Detecting the number of connection relationships between nodes in the directed graph; When the number of connection relationships is less than or equal to a second threshold, the directed graph is sent to a single processing unit for strongly connected component mining.

9. The method according to claim 8, It is characterized in that The method further comprises the following steps: When the number of nodes is greater than a first threshold or when the number of connection relationships is greater than a second threshold, the first label corresponding to each of the nodes in the updated directed graph is initialized, and the step of dividing the directed graph into multiple block graphs is returned.

10. The method according to claim 8, It is characterized in that The step of sending the directed graph to a single processing unit for strongly connected component mining includes: sending the directed graph to a single processing unit; In the processing unit, a depth-first search is performed on the directed graph, and each of the nodes is added to a stack in sequence according to the order in which they are searched; Searching for the fourth child node in sequence from the top of the stack to the rest of the stack; the first timestamp and the second timestamp corresponding to the fourth child node are the same; wherein the first timestamp is the timestamp when each of the nodes is searched, and the second timestamp is the earliest timestamp that can be traced back to each of the nodes; It is determined that nodes having the same first timestamp or second timestamp as the fourth child node belong to the same strongly connected component.

11. The method according to claim 1, It is characterized in that After all nodes in the directed graph are updated, determining that the nodes carrying the same first label as the first child node belong to the same strongly connected component includes: After all nodes in the directed graph are updated, determining the first child node from the directed graph; Starting from the first child node, a depth-first search is performed along the in-degree or out-degree direction to determine that nodes carrying the same first label as the first child node belong to the same strongly connected component.

12. A device for mining strongly connected components of graph data, It is characterized in that include: An acquisition module, used for acquiring a directed graph; the directed graph includes a plurality of nodes, each of the nodes corresponds to a first label, the first label includes a first in-degree attribute value and a first out-degree attribute value; different nodes correspond to different first in-degree attribute values, and different nodes correspond to different first out-degree attribute values; A partitioning module, used for partitioning the directed graph into a plurality of block graphs; A storage and update module, configured to store the first label of each node in the directed graph, perform asynchronous label propagation update on the first label of each node in the block graph according to the stored first label, and update the stored first label according to the result of the label propagation update; A processing module, configured to determine, after all nodes in the directed graph are updated, that nodes carrying the same first label as the first child node belong to the same strongly connected component; the first child node is a node having the same first label before and after the update; The storage and update module is specifically used for: storing the first in-degree attribute value of each of the nodes in the directed graph to a server; Asynchronously pulling the first in-degree attribute value of the node in each of the block graphs and the third in-degree attribute value corresponding to each of the nodes through multiple processing units; wherein each of the processing units processes at least one of the block graphs; the third in-degree attribute value is the first in-degree attribute value of the in-degree connection node of the node; In the processing unit, the first in-degree attribute value of the node is updated by the third in-degree attribute value corresponding to the node, and the updated first in-degree attribute value is synchronized to the server.

13. A computer device, It is characterized in that include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having a processor-executable program stored therein, Features: The processor-executable program is used to implement the method according to any one of claims 1 to 11 when executed by the processor.

15. A computer program product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images