A louvain-based complex network simple path discovery method

Abstract

The application discloses a Louvain-based complex network simple path discovery method, which comprises the following steps: firstly, data preprocessing is performed on node data and edge relationship data, and the processed data is used to establish a complex network; then, a community discovery is performed by using a Louvain algorithm, community labels are added to the discovered communities, and special nodes in the network after community division are marked to establish a community edge relationship network; the names of a starting node and a terminal node are inputted, and the community labels of the two nodes are acquired; a simple path discovery is performed in the community edge relationship network by using a tracking simple path search method, simple paths between communities are acquired, the paths between each pair of connected nodes in the acquired simple paths between communities are expanded and spliced to search the simple paths, and all simple path sets between the two nodes are outputted. The application solves the problem of slow speed in detecting all simple paths between two nodes in a complex network in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of complex network technology, specifically relating to a simple path discovery method for complex networks based on Louvain. Background Technology

[0002] In the analysis of complex networks, the analysis of simple paths between two nodes is often involved. A simple path is a path in which no two nodes repeat each other. In some applications, knowing whether a path exists between two nodes may be sufficient to solve the problem, but in other applications, it is necessary to know all simple paths between two nodes to better analyze the network and the relationships between nodes.

[0003] Finding all simple paths between two nodes is a typical NP-hard problem, with its complexity increasing dramatically with the size of the network. Most current algorithms for solving this problem are improvements on depth-first search, while others use methods such as stacks and priority queues. The former is based on a recursive function, while the latter is only suitable for very small networks and cannot handle large datasets well. Summary of the Invention

[0004] The purpose of this invention is to provide a simple path discovery method for complex networks based on Louvain, which solves the problem of slow speed in detecting all simple paths between two nodes in the existing technology.

[0005] The technical solution adopted in this invention is a simple path discovery method for complex networks based on Louvain, which is implemented according to the following steps:

[0006] Step 1: Perform data preprocessing on node data and edge relationship data. The nodes are the company names, the edge relationships are the holding relationships between companies, and the weight of the directed edge is the amount of capital subscribed by the company. The processed data is then used to build a complex network.

[0007] Step 2: Use the Louvain algorithm to discover communities, add community tags to the discovered communities, mark special nodes in the community-divided network, and establish a community edge relationship network.

[0008] Step 3: Input the names of the starting node and the ending node, obtain the community tags of the two nodes, use the community tags of the two nodes to perform simple path discovery in the community edge relationship network using the simple path tracing search method, and obtain the simple path between communities. If the input node name is incorrect or does not exist, repeat step 3. If no simple path between communities is found, end.

[0009] Step 4: Expand the path between each pair of connected nodes in the simple path between communities obtained in Step 3. Each expansion operation limits the network size to no more than two communities. In the network with the limited size, the simple path tracing search method is recursively called to discover simple paths. Store the set of paths found in each step. If no path exists or no path is found during the expansion process, the process ends.

[0010] Step 5: Concatenate the simple paths found in the search and output the set of all simple paths between the two nodes.

[0011] The invention is further characterized in that,

[0012] Step 1 is implemented in the following steps:

[0013] Step 1.1, Basic data preprocessing: Parse the node data and edge relationship data, and clean up redundant data;

[0014] Step 1.2: After data preprocessing, use Pandas to extract the names of all companies from the Excel file containing the data, create a company name index table, assign a unique index value to each company name, and replace the company names in the table with the index. Use Pandas to read the processed data file and import it into the database. First, read the company index to create nodes in the complex network, then read the edge relationships and subscribed capital to create edge relationships and assign weights to them, thus constructing the complex network.

[0015] Step 2 is implemented in the following steps:

[0016] Step 2.1: Use the Louvain algorithm for community detection.

[0017] Step 2.2: Extract the nodes within each community that are connected to other communities. The marked nodes will be referred to as special nodes in subsequent steps.

[0018] Step 2.3: Establish a network of inter-organizational relationships.

[0019] Step 2.1 is as follows:

[0020] Step 2.1.1: Initially, treat each node as a community, with the number of communities being the same as the number of nodes.

[0021] Step 2.1.2: Merge each node with its adjacent nodes in turn, calculate whether their modularity gain is greater than 0, and if it is greater than 0, put the node into the community of the adjacent node.

[0022] Step 2.1.3: Iterate the second step until the algorithm is stable, that is, the community to which all nodes belong no longer changes.

[0023] Step 2.1.4: Compress all nodes of each community into a single node. The weights of nodes within a community are converted into the weights of the new node, and the weights between communities are converted into the weights of the edges of the new node.

[0024] Step 2.1.5: Repeat steps 2.1.1-2.1.3 until the algorithm is stable.

[0025] Step 2.3 is as follows:

[0026] After step 2.1, the Louvain algorithm will assign a community label to each node. A small-scale network will be rebuilt in the database, where nodes represent communities and node names are community labels. If n communities are divided, n nodes will be established in the community edge relationship network. The edge relationships between communities are established by traversing the Excel spreadsheet storing the data and the enterprise name index table. These edge relationships are paths in the community edge relationship network, which reflects the directed connections between communities. This community edge relationship network is a compression of the overall scale of the complex network.

[0027] Step 3 is implemented in the following steps:

[0028] Step 3.1: When starting to query all simple paths between two companies, you need to enter the name of the starting company and the name of the ending company. After entering the name, you can find the index corresponding to the company name through the company name index table obtained in Step 1.2. If the entered node name is incorrect or the entered node name is not in the complex network, you need to re-enter the node name.

[0029] Step 3.2: If the input node names match the rules, the community tags of the two nodes are obtained. If the two nodes have the same community tag, the simple path tracing search method is called to search for all simple paths between the two nodes in the network of the communities they belong to and output the results. If no simple path exists, the process ends.

[0030] Step 3.3: If the community labels of the two nodes are inconsistent, the simple path search method is called to perform path search in the community edge relationship network established in step 2 through the two community labels to obtain a set of simple paths composed of community labels. If no simple path exists, the process ends.

[0031] Step 4 is implemented in the following steps:

[0032] Step 4.1: If Step 3 yields a simple path set composed of community tags, then the paths need to be expanded. Each path in the simple path set composed of community tags needs to be processed. Ultimately, the simple path set composed of community tags needs to be expanded into a simple path set composed of nodes within the community.

[0033] Step 4.2: The edge relationships of each path in the simple path set composed of community tags are expanded and stored in the network tree data structure. The network tree data structure is a set of nodes. This set is an empty set, or it can be composed of several different root nodes r1, r2...rm and 0 or more non-empty sub-network trees T1, T2...Tn.

[0034] Step 5 is implemented in the following steps:

[0035] Step 5.1: Check the connectivity of each path in the simple path set obtained in Step 4, and check for duplicate paths. The connectivity check is to check whether the tail node of one path is connected to the starting point of another path. If duplicate paths are found, the redundant parts will be deleted from the tree structure.

[0036] Step 5.2: Piece the path using the tree structure. After the connectivity check in step 5.1 is completed, the simple paths stored in the two nodes within the tree structure are pieced together according to their levels.

[0037] The beneficial effects of this invention are that it provides a fast and accurate simple path discovery method for complex networks based on Louvain. Unlike ordinary path detection methods, this invention significantly eliminates blind, large-scale path searches during path detection, transforming the logic of path detection from depth- or breadth-based searches to operations such as expansion and concatenation based on the correct direction. Since the computations related to path expansion and concatenation can be processed concurrently, multi-processing or multi-threading can be used in the algorithm implementation to further accelerate the path detection speed. Attached Figure Description

[0038] Figure 1 This is a flowchart of a simple path discovery method for complex networks based on Louvain, according to the present invention. Detailed Implementation

[0039] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0040] This invention discloses a simple path discovery method for complex networks based on Louvain, the flowchart of which is shown below. Figure 1 As shown, please follow these steps:

[0041] Step 1: Preprocess the node data and edge relationship data. In the data used in this invention, nodes represent company names, edge relationships represent the holding relationships between companies, and the weight of the directed edge is the subscribed capital contribution of the company. For example, if company A invests 12 million in company B, then node A pointing to node B will be generated in the network, and the weight of the edge relationship between nodes A and B will be 12 million. The processed data will then be used to build a complex network.

[0042] Step 1 is implemented in the following steps:

[0043] Step 1.1: Basic Data Preprocessing: Parse the node data and edge relationship data, and clean up redundant data. The original data contains various information, such as company name, unified social credit code, company type, operating status, primary industry, secondary industry, province, city, district / county, establishment date of the invested company, registered capital of the invested company, investment ratio, investment time, and subscribed capital. We need to extract the company name, investment time, and subscribed capital from this data. Here, we use the Pandas library in Python for data extraction. The subscribed capital contains multiple currency types, which need to be converted to RMB using a unified exchange rate. The converted subscribed capital needs to be normalized, i.e., all subscribed capital is converted to numbers between [0,1].

[0044] Cleaning redundant data involves removing duplicate edge relationships. If identical edge relationships appear (e.g., multiple investments by company A in company B), the investment time needs to be used to determine if it's a duplicate edge. The processed data is saved in an Excel file, with columns for investing company, invested company, and subscribed capital (normalized). If the dataset was already clean and cleaned initially, then that part...

[0045] Step 1.2: After data preprocessing, use Pandas to extract the names of all companies from the saved Excel file (because a company may appear multiple times in the table). Create a company name index table, assigning a unique index value to each company name, and replace the company names in the table with the index (because some company names are too long, so an index is used). Use Pandas to read the processed data file and import it into the database. First, read the company indexes to create nodes in a complex network. Then, read the edge relationships (read pairs of companies with investment relationships) and subscribed capital to create edge relationships and assign weights (the weights are the normalized subscribed capital) to construct the complex network.

[0046] Step 2: Use the Louvain algorithm to discover communities, add community tags to the discovered communities, mark special nodes in the community-divided network, and establish a community edge relationship network.

[0047] Step 2 is implemented in the following steps:

[0048] Step 2.1: Use the Louvain algorithm for community discovery. The Louvain algorithm is a community partitioning method based on modularity. The algorithm's process is as follows: 1. Initially, treat each node as a community, with the number of communities matching the number of nodes. 2. Merge each node with its adjacent nodes sequentially, calculating their modularity gain. If the gain is greater than 0, place the node into the community of its adjacent node. 3. Iterate through step 2 until the algorithm stabilizes, meaning the communities to which all nodes belong no longer change. 4. Compress all nodes in each community into a single node. The weights of nodes within a community are converted into the weights of the new node, and the weights between communities are converted into the weights of the edges of the new node. 5. Repeat steps 1-3 until the algorithm stabilizes. After the Louvain algorithm finishes running, each node will be assigned a community label, which will be added to the enterprise name index table (the output of the Louvain algorithm). For example, if node A is assigned to community C1, the attribute C1 will be added to node A in the database, indicating that node A belongs to community C1.

[0049] Step 2.2: Extract the nodes within each community that are connected to other communities. In Step 2.1, community tags were added to the database for each node in the complex network. If two nodes have an edge relationship but different community tag attributes, special tags will be added to these nodes. For example, if node A points to node B, and node A's community tag is C1, and node B's community tag is C2, then node A is a node within community C1 that is connected to other communities, and node B is a node within community C2 that is connected to other communities. Nodes like A and B will be uniformly assigned the attribute tag=1 in the database. These marked nodes are referred to as special nodes in subsequent steps.

[0050] Step 2.3: Establish an inter-community edge relationship network. After step 2.1, the Louvain algorithm assigns a community label to each node. Now, a smaller network needs to be rebuilt in the database. Nodes in this network represent communities, and their names are community labels. If n communities are defined, n nodes will be created in the inter-community edge relationship network. Edge relationships between communities are established by traversing the Excel spreadsheet storing company names and investment relationships, and the company name index table storing company name index information and community labels. These edge relationships are paths within the inter-community edge relationship network. For example, if the Excel spreadsheet shows that company A invests in company B, and the company name index table shows that company A belongs to community C1 and company B belongs to community C2, then a path from C1 to C2 will be added to the inter-community edge relationship network. This inter-community edge relationship network reflects the directed connections between communities and is a compression of the overall scale of a complex network.

[0051] Step 3: Input the names of the starting and ending nodes to obtain the community tags of the two nodes. Use the community tags of the two nodes to perform simple path discovery in the community edge relationship network using the simple path tracing search method (an algorithm improved from depth-first search) to obtain the simple path between communities. If the input node names are incorrect or do not exist, repeat step 3. If no simple path between communities is found, end the process.

[0052] Step 3 is implemented in the following steps:

[0053] Step 3.1: When starting to query all simple paths between two companies, you need to enter the name of the starting company and the name of the ending company (the names of the two companies are selected and entered according to the query needs of the actual application scenario). After entering the name, you can find the index corresponding to the company name through the company name index table obtained in Step 1.2. If the entered node name is incorrect or the entered node name is not in the complex network, you need to re-enter the node name.

[0054] Step 3.2: If the input node names match the rules, the community labels of the two nodes are obtained. If the two nodes have the same community label, the simple path tracing search method is called to search for all simple paths between the two nodes in the network of their respective communities (not in the complete complex network) and output the results. Not searching for paths in the complete complex network significantly reduces the time required for path searching. The algorithm for searching all simple paths between the two nodes is implemented using the simple path tracing search method. This algorithm is a direct solution for finding paths, based on depth-first search and a recursive function. It checks each edge vt attached to v to determine if there is a simple path from t to w that does not pass through v. It uses a vertex index array to label v, so that during the recursive call, paths passing through v are not checked; if no simple path exists, the process ends.

[0055] Step 3.3: If the community labels of the two nodes are inconsistent, the simple path search method is called (mentioned in step 3.2). In the community edge relationship network established in step 2, the path search is performed through the two community labels to obtain a set of simple paths composed of community labels. For example, C1→C3→C5 represents the path between community C1 and community C5. If no simple path exists, the process ends.

[0056] Step 4: Expand the path between each pair of connected nodes in the simple path between communities obtained in Step 3. Each expansion operation limits the network size to no more than two communities. In the network with the limited size, the simple path tracing search method is recursively called to discover simple paths. Store the set of paths found in each step. If no path exists or no path is found during the expansion process, the process ends.

[0057] Step 4 is implemented in the following steps:

[0058] Step 4.1: If Step 3 yields a simple path set composed of community tags, then the paths need to be expanded. Each path in the simple path set composed of community tags needs to be processed. Ultimately, the simple path set composed of community tags needs to be expanded into a simple path set composed of nodes within the community. During expansion, special nodes within the community (nodes with attribute tag=1, mentioned in Step 2.2) need to be queried. These special nodes will be used as start or end points to perform simple path search using the tracking simple path search method. Each call to the tracking simple path search method will perform a simple path search in the network composed of the two communities.

[0059] Step 4.2: The expanded edge relationships of each path in the simple path set composed of community tags are stored in a network tree data structure. The network tree data structure is a set of nodes, which can be empty or composed of several different root nodes r1, r2…rm and 0 or more non-empty sub-network trees T1, T2…Tn. The root of each of these sub-network trees has at least one edge connected to the root node ri of the network tree, where 1≤m, 1≤n, and 1≤i≤n. First, the network tree data structure is created, and each expansion operation is stored in the same level of the network tree. For example, the first expanded path set is stored in the first level of the network tree (the 0th level is the node corresponding to the starting company name), the second expanded path set is stored in the second level of the network tree, and so on. This step facilitates the subsequent concatenation of simple paths.

[0060] Step 5: Concatenate the simple paths found in the search and output the set of all simple paths between the two nodes.

[0061] Step 5 is implemented in the following steps:

[0062] Step 5.1: Check the connectivity of each path in the simple path set obtained in Step 4, and check for duplicate paths. The connectivity check is to check whether the tail node of one path is connected to the starting point of another path. If duplicate paths are found, the redundant parts will be deleted from the tree structure.

[0063] Step 5.2: Concatenate paths using a tree structure. After the connectivity check in Step 5.1, concatenate the simple paths stored in two nodes within the tree structure according to their hierarchy. During the concatenation process, the successor nodes of the tree nodes need to be checked. When concatenating hierarchically, a tree node can only be used for simple path concatenation if it is connected to the node at the next lower level, to ensure the correctness of the path concatenation result.

[0064] Example:

[0065] The dataset used in this example is a complex network constructed from data extracted from a corporate equity penetration diagram using visual recognition methods. This network has 12,505 nodes, representing 12,505 companies; and 13,316 directed edges, representing the shareholding relationships between companies. Due to the nature of the dataset, this complex network contains a large number of branches and loops, meeting the network complexity requirements of general complex network datasets. Python has a dedicated package, Networkx, for handling complex networks, with a simple path search function called all_simple_path. In comparative experiments, the time consumed by the all_simple_path function to identify all simple paths between two nodes is 300-400 times longer than the time consumed by this invention to identify all simple paths between the same two nodes. Therefore, the all_simple_path function was not used in subsequent related experiments.

[0066] After using the Louvain algorithm for community detection in a complex network, the entire complex network is divided into communities. If the complex network is divided into n communities, n community labels will be generated. The labeling information of special nodes will appear in the structure of these nodes. Four test cases, both general and special, are tested: Case 1, two nodes are between adjacent communities; Case 2, two nodes are across multiple communities; Case 3, two nodes are within the same community; Case 4, no path exists. To verify the accuracy of path detection, the paths detected by this invention are compared with the actual paths in the above four test cases, as shown in Table 1. The comparison results are as follows:

[0067] Table 1 Comparison Results of Four General and Special Cases

[0068] Test cases Actual paths (number of paths) Paths detected. accuracy Case 1 use case 2 2 precise Case 2 use case 4 4 precise Case 3 use case 1 1 precise Case 4 use case 0 0 precise

[0069] Table 2 shows the time consumed by the test cases in the above four scenarios using the path detection method described in this invention, as detailed below:

[0070] Table 2 shows the time consumed by test cases in the above four scenarios using the path detection method described in this invention:

[0071]

[0072]

[0073] This invention uses the Louvain algorithm to discover communities in complex networks, compressing the complex network as a whole to find simple paths between communities. It then limits the size of the query network, ensuring that each query performs path searches within a network of at most two communities, thus avoiding the problem of blindly searching the entire network and the unnecessary system overhead of large-scale searches in complex networks.

Claims

1. A simple path discovery method for complex networks based on Louvain, characterized in that, The specific steps are as follows: Step 1: Perform data preprocessing on node data and edge relationship data. The nodes are the company names, the edge relationships are the holding relationships between companies, and the weight of the directed edge is the amount of capital subscribed by the company. The processed data is then used to build a complex network. Step 1 is implemented in the following steps: Step 1.1, Basic data preprocessing: Parse the node data and edge relationship data, and clean up redundant data; Step 1.2: After data preprocessing, use Pandas to extract the names of all companies from the Excel file containing the data, create a company name index table, assign a unique index value to each company name, and replace the company names in the table with the index. Use Pandas to read the processed data file and import the data file into the database. First, read the company index to create nodes in the complex network, then read the edge relationships and subscribed capital to create edge relationships and assign weights to them, thus constructing the complex network. Step 2: Use the Louvain algorithm to discover communities, add community tags to the discovered communities, mark special nodes in the community-divided network, and establish a community edge relationship network. Step 2 is implemented in the following steps: Step 2.1: Use the Louvain algorithm for community detection. Step 2.1 is as follows: Step 2.1.1: Initially, each node is treated as a community, and the number of communities is the same as the number of nodes; Step 2.1.2: Merge each node with its adjacent nodes in turn, calculate whether their modularity gain is greater than 0, and if it is greater than 0, put the node into the community of the adjacent node. Step 2.1.3: Iterate through the second step until the algorithm is stable, meaning that the communities to which all nodes belong no longer change; Step 2.1.4: Compress all nodes of each community into a single node. The weights of the nodes within a community are converted into the weights of the new node, and the weights between communities are converted into the weights of the edges of the new node. Step 2.1.5: Repeat steps 2.1.1-2.1.3 until the algorithm is stable; Step 2.2: Extract the nodes within each community that are connected to other communities. The marked nodes will be referred to as special nodes in subsequent steps. Step 2.3: Establish a network of inter-organizational relationships; Step 2.3 is as follows: After step 2.1, the Louvain algorithm will assign a community label to each node. A small-scale network will be rebuilt in the database, where nodes represent communities and node names are community labels. If n communities are divided, n nodes will be established in the community edge relationship network. The edge relationships between communities will be established by traversing the Excel spreadsheet storing the data and the enterprise name index table. These edge relationships are paths in the community edge relationship network, which reflects the directed connections between communities. This community edge relationship network is a compression of the overall scale of the complex network. Step 3: Input the names of the starting node and the ending node, obtain the community tags of the two nodes, use the community tags of the two nodes to perform simple path discovery in the community edge relationship network using the simple path tracing search method, and obtain the simple path between communities. If the input node name is incorrect or does not exist, repeat step 3. If no simple path between communities is found, end. Step 3 is implemented in the following steps: Step 3.1: When starting to query all simple paths between two companies, you need to enter the name of the starting company and the name of the ending company. After entering the name, you can find the index corresponding to the company name through the company name index table obtained in Step 1.

2. If the entered node name is incorrect or the entered node name is not in the complex network, you need to re-enter the node name. Step 3.2: If the input node names match the rules, the community tags of the two nodes are obtained. If the two nodes have the same community tag, the simple path tracing search method is called to search for all simple paths between the two nodes in the network of the communities they belong to and output the results. If no simple path exists, the process ends. Step 3.3: If the community labels of the two nodes are inconsistent, the simple path search method is called to perform path search in the community edge relationship network established in step 2 through the two community labels to obtain a set of simple paths composed of community labels. If no simple path exists, the process ends. Step 4: Expand the path between each pair of connected nodes in the simple path between communities obtained in Step 3. Each expansion operation limits the network size to no more than two communities. In the network with the limited size, the simple path tracing search method is recursively called to discover simple paths. Store the set of paths found in each step. If no path exists or no path is found during the expansion process, the process ends. Step 4 is implemented in the following steps: Step 4.1: If Step 3 yields a simple path set composed of community tags, then the paths need to be expanded. Each path in the simple path set composed of community tags needs to be processed. Ultimately, the simple path set composed of community tags needs to be expanded into a simple path set composed of nodes within the community. Step 4.2: The edge relationships of each path in the simple path set composed of community tags are expanded and stored in the network tree data structure. The network tree data structure is a set of nodes. This set is an empty set, or it can be composed of several different root nodes r1, r2...rm and 0 or more non-empty sub-network trees T1, T2...Tn. Step 5: Concatenate the simple paths found in the search and output the set of all simple paths between the two nodes; Step 5 is implemented in the following steps: Step 5.1: Check the connectivity of each path in the simple path set obtained in Step 4, and check for duplicate paths. The connectivity check is to check whether the tail node of one path is connected to the starting point of another path. If duplicate paths are found, the redundant parts will be deleted from the tree structure. Step 5.2: Piece the path using the tree structure. After the connectivity check in step 5.1 is completed, the simple paths stored in the two nodes within the tree structure are pieced together according to their levels.

Images