A distributed critical path mining method for billion-level enterprise transaction networks
By combining the graph attention network and random walk method, using GAT to calculate the attention weights between nodes and compute in parallel in a distributed processing framework, the efficiency and interpretability problems of critical path identification in a billion-level enterprise transaction network are solved. Efficient and accurate analysis results and decision-making support are achieved.
Patent Information
- Application Number
- CN202411298771.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-18
AI Technical Summary
In the enterprise trading network with hundreds of millions of nodes, traditional graph analysis methods and random walk methods are difficult to efficiently identify key transaction paths due to their high computational complexity and lack of scalability. The lack of clear guidance on path selection, which reduces the availability of analysis results Explanatory.
Combining the graph attention network (GAT) and random walk method, the random walk process is guided by calculating the attention weight between nodes in each subgraph, identifying the transaction influence between enterprises, and the distribution in Spark and Pandas UDFs parallel computing in the processing framework improves the objective and interpretability of path selection.
Significantly improve the identification accuracy and analysis efficiency of critical paths in large-scale enterprise trading networks, enhance the logic and interpretability of path selection, support more accurate and understandable enterprise transaction data analysis, and help make decisions in complex networks and trading strategy optimization.
Smart Images

Figure CN118820534B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and in particular relates to a method for mining distributed critical paths of billion-level enterprise transaction networks by combining a graph attention network (GAT) and a random walk method. Background Art
[0002] In an enterprise transaction network, enterprises can be regarded as nodes, and transactions constitute the edges between nodes. As the scale of enterprise transaction networks continues to expand, such networks often contain huge amounts of data. Traditional graph analysis methods face huge challenges in processing large-scale data. When the number of nodes reaches the billion level, the computational complexity usually reaches O(n²), which seriously affects processing efficiency. At the same time, in a complex enterprise transaction network, it becomes very difficult to identify and analyze key transaction paths, and the traditional random walk method lacks clear guidance on path selection, which reduces the interpretability of the analysis results.
[0003] Graph Attention Network (GAT) shows significant advantages in capturing the importance of relationships between nodes, while random walk is an effective method for exploring graph structure. Random walk is a commonly used method in graph analysis that can effectively explore the structure and relationships in the graph. However, traditional random walk methods often lack clear guidance and interpretability in path selection, which is particularly insufficient in enterprise transaction network analysis, and understanding the importance of transaction relationships between enterprises is crucial for decision support. Therefore, how to organically combine these two technologies to improve the efficiency and accuracy of key transaction path analysis in large-scale enterprise transaction networks, especially in terms of improving scalability and interpretability, is still an urgent problem to be solved. Summary of the invention
[0004] In the scenario of tens of millions or even hundreds of millions of nodes, the traditional graph analysis methods (frequent subgraph mining) and graph neural networks (GNN) used in the current key transaction path identification technology of enterprise transaction networks are often difficult to process these data efficiently due to their high computational complexity, usually NP, i.e., non-deterministic polynomial problems, and lack of scalability; and the traditional random walk method often lacks clear guidance and explainability in path selection, and lacks support for the importance of understanding the transaction relationship between enterprises. In response to these problems, the present invention proposes a distributed key path mining method for enterprise transaction networks at the billion level, which uses GAT to capture and quantify the importance relationship between nodes, obtain the transaction influence between enterprises, guide the random walk process to focus more on the key transaction path, and realize the identification of the key path of large-scale enterprise transaction networks.
[0005] The present invention provides a distributed critical path mining method for a billion-level enterprise transaction network, comprising the following steps:
[0006] Step 1: Obtain an isomorphic graph of the enterprise transaction network to be studied. Each node in the graph represents an enterprise. When there is a transaction setting edge between enterprises, split the subgraph with each enterprise node as the center; the subgraph of each enterprise node contains the first-order neighbor nodes and second-order neighbor nodes of the node;
[0007] In step 2, each subgraph is loaded into a Pandas UDF independently, and multiple subgraphs will be executed in parallel.
[0008] Step 3: Use the graph attention network in each Pandas UDF to calculate the attention weight of the subgraph and obtain the aggregated features of the central node of the subgraph. Set up a multi-layer graph attention network structure in each Pandas UDF. According to the transaction relationship in the subgraph, first aggregate the transaction impact of the second-order neighbor nodes to the features of the first-order neighbor nodes, and then aggregate the features of the first-order neighbor nodes to the features of the central node, and calculate the attention weight of the central node to each first-order neighbor.
[0009] Step 4: After the GAT attention weights of all subgraphs are calculated, an attention-guided random walk is performed;
[0010] For each node, select the top 10 first-order neighbor nodes with the largest attention weights, store the node's attention weights to the 10 first-order neighbor nodes in a matrix dictionary, and cache it in Spark; during random walks, the selection of nodes on the path depends on the attention weights cached in Spark;
[0011] Step 5: Obtain M paths through random walks, score the importance of each path, and identify the key paths in the enterprise transaction network; M is greater than 2; the importance score of each path is obtained by calculating the product of the attention weights of all connected node pairs on the path;
[0012] Step 6, calculate the node influence by measuring the average importance score of the node in all possible paths and identify the key nodes in the enterprise transaction network.
[0013] Compared with the prior art, the advantages and positive effects of the present invention are:
[0014] (1) The method of the present invention combines the GAT and random walk methods to obtain the critical path, so that the path selection of the random walk no longer depends solely on the topological structure of the graph, but adds the importance evaluation of the transaction relationship between nodes. Each step of the random walk is based on the transaction influence between the actual nodes (enterprises). This improves the target and interpretability of the random walk in the enterprise transaction network, thereby improving the logic and interpretability of the path selection, realizing efficient analysis of large-scale enterprise transaction networks, and improving the accuracy of critical path identification.
[0015] (2) In order to implement the method of the present invention on large-scale graph data with hundreds of millions of nodes, the method of the present invention adopts Spark and Pandas UDF technology. This technology enables the present invention to efficiently divide the entire enterprise transaction network into multiple sub-graphs, and independently apply GAT to each sub-graph to calculate the attention weight, thereby realizing distributed parallel processing and expanding the ability to process large-scale data. This not only significantly improves the data processing speed and efficiency, but also ensures the accuracy and practicality of the calculation results.
[0016] (3) The method of the present invention uses GAT to capture and quantify the transaction influence between enterprise nodes, guiding the random walk process to focus more on key transaction paths. It not only improves the accuracy of path selection, but also enhances the model's ability to explain important transaction relationships between enterprises. It can provide more accurate and understandable analysis results when processing huge amounts of enterprise transaction data, support the making of key decisions in complex networks, and help optimize transaction strategies and relationship management between enterprises. BRIEF DESCRIPTION OF THE DRAWINGS
[0017] Figure 1 It is a flow chart for implementing the distributed critical path mining method for billion-level enterprise transaction networks of the present invention;
[0018] Figure 2 is an example of a simple subgraph. DETAILED DESCRIPTION
[0019] The technical solution of the present invention is described below in conjunction with the accompanying drawings and embodiments.
[0020] The distributed critical path mining method and device of the billion-level enterprise transaction network of the present invention are applied in large-scale enterprise transaction networks. First, the isomorphic graph data of the enterprise transaction network is obtained. Each node in the isomorphic graph represents an enterprise, and the edge in the graph represents the transaction between two enterprises. The characteristics of the enterprise nodes are obtained according to the needs. The node characteristics mainly include financial business characteristics, such as transaction amount, transaction frequency, credit score, risk transaction amount, etc. According to the application scenario requirements, for example, if a credit detection is performed, the characteristics about the enterprise credit are set more. If a fraud risk detection is performed, the characteristics of fraud are set more. The node attribute characteristics can be adjusted and set as needed. Then execute the method or device of the present invention to extract the critical path.
[0021] like Figure 1 As shown, the distributed critical path mining method for billion-level enterprise transaction networks according to an embodiment of the present invention includes the following steps 1 to 6.
[0022] Step 1: split the isomorphic graph of the enterprise transaction network to obtain a subgraph centered on each node.
[0023] First, the isomorphic graph of the entire enterprise transaction network is split into subgraphs centered on each enterprise node. For example, for isomorphic graph data of 10 million nodes, a total of about 10 million subgraphs are split. Each subgraph includes a node and its first-order and second-order neighbor nodes and all the edges of these nodes. Through this splitting strategy, the present invention localizes the calculation of the attention mechanism to each node and its direct neighbors, effectively reducing the computational complexity and making the processing of each subgraph more efficient and manageable.
[0024] like Figure 1 As shown, the enterprise transaction network graph with N nodes is split into N sub-graphs.
[0025] Step 2: Load each subgraph into a Pandas UDF independently and use Apache Spark and Pandas UDF technology to implement distributed parallel processing of the subgraphs.
[0026] Apache Spark is an open source computing distribution system. Pandas UDF (User Defined Function) is a special function in Spark that allows efficient data processing using the Pandas library in Spark. Pandas UDF combines the ease of use of Pandas with the distributed computing capabilities of Spark, making it more efficient to perform complex data operations on large-scale data sets. Each subgraph is independently loaded into a Pandas UDF, which is responsible for the calculation of the subgraph. This method takes advantage of Spark's distributed computing framework and can handle a large number of subgraph calculation tasks at the same time, significantly improving the overall processing speed.
[0027] Step 3, calculate the GAT attention weights of the loaded subgraphs in each Pandas UDF.
[0028] In each Pandas UDF, the attention coefficient of each node to its neighbors is calculated by applying the graph attention network GAT. The attention coefficient is calculated using the following formula.
[0029] For a subgraph G=(V,E), V represents the node set of the subgraph, E represents the edge set of the subgraph, and the neighbor set of node i is represented by N i , use GAT to calculate the attention weights of nodes in subgraph G, the attention weights of node i to neighbor node j The calculation is as follows:
[0030]
[0031] in, represents the similarity coefficient between node i and node k, Represents the similarity coefficient between node i and node j. The calculation is as follows:
[0032]
[0033] Among them, LeakyReLU is the activation function, a is the trainable attention vector, W is the trainable weight matrix, , They represent the features of node i and node j respectively, || represents the concatenation operation, and the superscript T represents the transposition.
[0034] The embodiment of the present invention focuses on detecting the credit of an enterprise, and the node features set include at least the enterprise's transaction amount, transaction frequency, credit score and risk transaction amount. The GAT calculates the attention weights between nodes in the subgraph, thereby capturing and quantifying the transaction influence between nodes. By calculating the similarity of node features, enterprises with similar transaction features can be identified. By learning the attention vector a, the GAT model can adaptively adjust the importance of different features to determine which node feature combinations are more important in deciding transaction influence.
[0035] Specifically, for each subgraph, not only the attention weights of the central node to its direct neighbors are calculated, but a wider range of influences are captured through a multi-layer GAT structure. In the case of a second-order subgraph, the calculation of attention weights and the propagation of information are performed as follows:
[0036] First-level aggregation: The influence of direct neighbor nodes (first-order neighbors) on the central node is calculated and aggregated;
[0037] Second-level aggregation: transaction information of second-order neighbors is transmitted to the central node through first-order neighbors, capturing indirect influence;
[0038] Final aggregation: The central node combines the information from the first-order and second-order neighbors to form the final node representation.
[0039] This process can be described as three aggregation operations: first, based on the transaction relationship between the second-order neighbors and the first-order neighbors, the transaction influence of the second-order neighbors is aggregated to the characteristics of the first-order neighbors; then, based on the transaction relationship between the first-order neighbors and the central node, the transaction influence of the first-order neighbors is aggregated to the central node. Finally, the attention weight of the central node to each first-order neighbor and the characteristics of the transaction influence of the first-order neighbors and the second-order neighbors aggregated by the central node are obtained.
[0040] like Figure 2As shown, it is a simple subgraph example, in which the blue node represents the central node, and the direction of the directed edge represents the flow of capital transactions. As can be seen from the figure, although it is a second-order subgraph of the central node, the transaction path nodes may exceed the second order. When calculating the GAT attention weight of the subgraph, only the attention weights between directly connected nodes are calculated, and the topological structure information of the network is implicitly used through GAT, reflecting the direct business relationship in the actual transaction network, that is, the present invention calculates the attention weight of the node in this way, not only considering the node characteristics, but also considering the relative position of the node in the network structure.
[0041] It is worth noting that although the present invention uses a second-order subgraph, the transmission of transaction information may exceed the second order. This is because in each layer of aggregation, nodes are synthesizing information about their neighbors, which in turn contain information about their own neighbors. Therefore, even in a second-order subgraph, the central node may indirectly receive information from nodes that are more than a second-order distance away.
[0042] Through multi-level attention mechanisms and information propagation, and through multi-level aggregation, the present invention enables GAT to effectively capture not only the relationships between directly connected nodes, but also the influence between more distant nodes, and to more comprehensively understand the complex network structure and more remote transaction relationships in the network, thereby providing more accurate and insightful results in enterprise transaction network analysis.
[0043] Step 4: After completing the GAT attention weight calculation for each node, perform an attention-guided random walk.
[0044] In order to optimize storage and improve the execution efficiency of random walks, the present invention also adopts an efficient data storage and access strategy. The embodiment of the present invention only saves the attention weights of the top 10 neighbor nodes of each node. These attention weights are stored in a matrix dictionary and cached in Spark. Such processing not only reduces storage requirements, but also ensures that the random walk algorithm can quickly access these key data during operation. The number of neighbor node attention weights stored for each node is preferably set to 10.
[0045] During the random walk, the path selection depends on these pre-calculated and stored attention weights. Specifically, the probability of transferring from one node to another is determined based on the attention weight between them. The higher the weight of the edge, the greater the probability of being selected by the random walk. This attention-based random walk strategy effectively utilizes the pre-calculated graph structure information, making the random walk more focused on important and information-rich paths in the graph.
[0046] The implementation method is as follows:
[0047] At the beginning of the random walk, any node is selected as the starting point. At each step, the probability of transferring to each reachable neighbor node is calculated based on the stored attention weight of the current node.
[0048] Using the attention weights calculated by GAT, redefine the transition probability from node i to j as follows:
[0049]
[0050] in, It represents the set of 10 neighbor nodes with the largest attention weights selected by node i. In this way, the edge with higher attention weight has a greater probability of being selected by random walk.
[0051] Step 5: Score the path importance and identify the critical paths in the enterprise transaction network.
[0052] After random walk, multiple paths are obtained. This step is used to evaluate the importance of the path selected during the random walk. Let path p=(v1,v2,...,v L ), contains L nodes, the importance score Score(p) of path p is defined as the product of the attention weights of all consecutive node pairs on the path, as follows:
[0053]
[0054] in, Representation Node For Node The attention weight.
[0055] Step 6: Calculate node influence and identify key nodes in the enterprise transaction network.
[0056] In the present invention, the influence of a node is evaluated by measuring the average importance of the node in all possible paths. This evaluation method can reveal the centrality and influence of a node in the entire network, which is particularly important for identifying key nodes and their role in the network. For a node v, the present invention calculates its average influence in all paths through the following two steps:
[0057] Step 6.1: For each node, accumulate the importance scores of all paths involved in the node.
[0058] For a node v, calculate the cumulative value of the importance scores of all paths involving the node. All paths involving node v include all paths with v as the starting point or end point, as well as all paths with v as an intermediate node. Then, use the path importance scoring formula to calculate the importance score of each path involved, and add them up to get the cumulative value.
[0059] Step 6.2: Calculate node influence. For a node v, node influence is defined as the sum of importance scores of all paths involving node v divided by the number of paths. The influence of node v The calculation is as follows:
[0060]
[0061] Where P represents the set of all paths involving node v, Represents the number of paths in set P.
[0062] The method of the present invention not only accurately reflects the importance of each node in the local subgraph, but also greatly shortens the processing time through parallel processing. Thanks to this technical implementation method, the present invention effectively adapts to the processing requirements of large-scale graph data and provides an efficient and accurate graph analysis tool. This method enhances the efficiency and targeting of random walks, making it more effective in applications such as social network analysis and recommendation systems. In addition, the calculation of path importance scores and node influence not only helps understand complex network structures, but also supports key decision-making processes, especially when identifying and optimizing key nodes and paths in the network.
[0063] Except for the technical features described in the specification, all of them are known technologies to those skilled in the art. The present invention omits the description of known components and known technologies to avoid redundancy and unnecessary limitation of the present invention. The implementation methods described in the above embodiments do not represent all implementation methods consistent with the present application. On the basis of the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative labor are still within the protection scope of the present invention.
Claims
1. A distributed critical path mining method for billion-level enterprise transaction networks, characterized in that: The steps include: Step 1: Obtain an isomorphic graph of the enterprise transaction network to be studied. Each node in the graph represents an enterprise. When there is a transaction setting edge between enterprises, split the subgraph with each enterprise node as the center; the subgraph of each enterprise node contains the first-order neighbor nodes and second-order neighbor nodes of the node; In step 2, each subgraph is loaded into a Pandas UDF independently, and multiple subgraphs will be executed in parallel. Step 3: Use the graph attention network in each Pandas UDF to calculate the attention weight of the subgraph and obtain the aggregated features of the central node of the subgraph; A multi-layer graph attention network structure is set in each Pandas UDF. Based on the transaction relationship in the subgraph, the transaction impact of the second-order neighbor nodes is first aggregated to the features of the first-order neighbor nodes, and then the features of the first-order neighbor nodes are aggregated to the features of the central node. The attention weight of the central node to each first-order neighbor node is calculated. Step 4: After the GAT attention weights of all subgraphs are calculated, an attention-guided random walk is performed; For each node, select the top 10 first-order neighbor nodes with the largest attention weights, store the node's attention weights to the 10 first-order neighbor nodes in a matrix dictionary, and cache it in Spark; During random walk, the selection of nodes on the path depends on the attention weights cached in Spark; Step 5: Obtain M paths through random walks, score the importance of each path, and identify the key paths in the enterprise transaction network, where M is greater than 2; the importance score of each path is obtained by calculating the product of the attention weights of all connected node pairs on the path; Step 6, calculate the node influence by measuring the average importance score of the node in all involved paths and identify the key nodes in the enterprise transaction network.
2. The method according to claim 1, characterized in that In the step 1, the node features are set to include the enterprise's transaction amount, transaction frequency, credit score and risky transaction amount.
3. The method according to claim 1 or 2, characterized in that: In step 4, when performing random walk, an enterprise node is initially randomly selected as the starting point. At each step, the node transfer probability is calculated based on the stored attention weights from the current node to the 10 first-order neighbor nodes, and the next node selection is performed; Calculate the transition probability from the current node i to the neighboring node j as follows: ;in, Represents the set of 10 first-order neighbor nodes with the largest attention weights selected by node i; is the attention weight of node i to neighbor node j, is the attention weight of node i to its neighbor node k.
4. The method according to claim 1 or 2, characterized in that: In step 5, assume that a path p contains L nodes, represented by p=(v1,v2,...,v L ), then the importance score Score(p) of path p is calculated as follows: ;in, Representation Node For Node The attention weight.
5. The method according to claim 1 or 2, characterized in that: In step 6, the influence of node v is calculated as follows: First, count the cumulative importance scores of all paths involved in node v, and then divide it by the number of paths involved to get the influence of node v. , the calculation formula is as follows: ; Where P represents the set of all paths involved in node v, represents the number of paths in the set P, Represents the importance score of path p.
Citation Information
Patent Citations
Enterprise valuation method and system based on heterogeneous graph neural network
CN114186799A
Multi-layer heterogeneous network node importance evaluation method based on attention mechanism
CN115658971A
Marketing topic key user identification method and device and computer equipment
CN118069936A
Method of analyzing social influence between internet forums and apparatus for the same
US20240202839A1
