Efficient financial knowledge graph division method

Abstract

The invention discloses an efficient financial knowledge graph division method. According to the method, an edge set and a vertex set are initialized, Degree Biased Sampling is carried out on edge data, and efficient partitioning of financial knowledge graph data is realized in combination with a gain-based partitioning algorithm. According to the method, the number of cross-partition edges is greatly reduced while the structural representativeness of the sampling graph is ensured, and the load balance and the distributed storage performance are improved. Experiments show that the edge-tangent ratio, the load balance and the operation efficiency of the method in large-scale financial knowledge graph data processing are superior to those in the prior art.

Description

Technical Field

[0001] The present invention relates to the field of large-scale knowledge graph data processing, and specifically to a distributed graph partitioning method based on representative subgraphs, which is used to improve the partitioning efficiency of ultra-large-scale graphs, reduce memory consumption, and optimize the load balancing and communication performance of distributed systems. Background Art

[0002] Financial knowledge graphs, a technology based on artificial intelligence and machine learning, establish relationships and hierarchies in financial knowledge using graph data. They are currently widely used in financial management and decision-making. In practical applications of financial knowledge graphs, to improve processing efficiency, large-scale graphs need to be partitioned into multiple subgraphs and assigned to different storage or computing nodes to enable parallel computing. However, graph partitioning is an NP-hard problem. Its optimization objective typically involves reducing the number of edges across partitions (edge-cut ratio) to reduce communication overhead while ensuring load balancing across partitions. Existing graph partitioning methods can be categorized as follows: Multi-level partitioning algorithms: These algorithms gradually simplify a graph through multiple coarsening and refinement steps, then partition the simplified graph. While these methods can achieve good partitioning quality, they require multiple loadings of the original graph and repeated adjustments to the partitions at each level. This leads to high memory and computational overhead, making them difficult to scale to very large graphs. Hashing and greedy heuristic methods: These methods reduce runtime by quickly allocating vertices and edges, but often come at the expense of a high edge-cut ratio, significantly increasing communication overhead and exhibiting poor load balancing. Graph sampling techniques: Generate a small subset of a graph through random sampling or specific rules to approximate the original graph. However, existing sampling methods are insufficient in preserving the structural characteristics of the original graph, which can easily lead to the partitioning results deviating from the global optimum. As the graph size continues to grow, the limitations of these methods in terms of efficiency, memory requirements, and partitioning quality become increasingly significant. Therefore, developing an efficient, memory-friendly, and high-quality graph partitioning method has become an important research direction in the field of distributed financial knowledge graph computing. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this paper proposes an efficient method for partitioning financial knowledge graphs. This method uses degree-biased sampling (DBS) to generate representative subgraphs and applies a gain-based partitioning algorithm to these subgraphs. This method can partition large-scale graphs with only two disk scans, significantly reducing memory requirements and optimizing partitioning quality. It also offers high efficiency, robustness, and scalability.

[0004] The technical solutions adopted by the present invention are as follows:

[0005] Step 1: Initialize the sampling set E S and the vertex set V S ; The specific steps are as follows:

[0006] Step 1.1: From the edge sequence E seq Load the initial edge set and read E seq The first ρ edges in the middle, and each edge e=(u,v) is stored in the sampling set E in the format of (e,w,r,k) S , where w is the weight, r is the random value, and k is the key value (w, r, k are initially 0);

[0007] Step 1.2: Update the vertex set V S , for the two vertices u and v of edge e, update the information according to their degree values. The calculation formula is: V S =V S ∪(u,V S [u].d+1)∪(v,V S [v].d+1);

[0008] Step 1.3: Calculate E S The weight and key value of the middle edge, traversing E S For each edge e=(u,v), calculate its attributes according to the following rules r = random(0,1), k = r 1 / w . Used to calculate the sampling set E S The weight w, random value r and key value k of each edge e=(u,v) in each edge are used to update the sampling set E in the next step S .

[0009] Step 2: Dynamically update the sampling set E based on degree biased sampling (DBS) S ;

[0010] Step 2.1: From the edge sequence E seq Starting at position ρ+1+i·η, read the next η edges into the temporary set E′, where i=0,1,2,... represents the index of the current batch;

[0011] Step 2.2: Update the vertex set V according to each edge e = (u, v) in E′ S Vertex degree information, recalculate E S The key value k of each element in , and set the element with the smallest key value as the current minimum element T;

[0012] Step 2.3: Dynamically update the sampling set E according to the information of each edge e = (u, v) in E′ S ; For the sampling set E S To update, the specific steps are as follows:

[0013] Step 2.3.1: Calculate the weight w, random value r, and key value k for each edge e = (u, v) in E′ using the following formula: r = random(0,1), k = r 1 / w ;

[0014] Step 2.3.2: Replacement sampling set E S Elements in: Decide whether to replace E based on the key value k of each edge e=(u,v) calculated in step 2.3.1 S If k>Tk, replace E with the current edge (e,w,r,key) S The element T with the smallest key value in the array is updated to E. S The new minimum key value element in, otherwise, continue to judge the next edge, always keeping E S The elements with large weight w are stored in .

[0015] Step 3: Repeat step 2 until the edge sequence E seq All edges in are processed completely, and the final ones retained in the sampling set E are used. S The edges in the graph represent the subgraph G S =(V S ,E S ).

[0016] Step 4: For the representative subgraph G S Apply the gain-based partitioning algorithm to divide it into k subgraphs, denoted as Divide each subgraph Save to the corresponding storage node.

[0017] Step 5: Process E seq The unsampled vertices and edges in the , according to the vertex information of each unsampled edge, save the edge to the corresponding edge set. seq The specific strategy for unsampled edges and vertices is: for E seq For each unsampled edge e=(u,v), if its vertex u or v is not in the vertex set V S In the example, it is assigned to a storage node i according to its locality. If vertices u and v belong to different partitions (i.e., i≠j, where u∈V i s and ) marks the edge as a "cross-partition edge (cutedge)" and saves it to the two partitioned edge sets ( and ); If vertices u and v belong to the same partition, store the edge in the edge set of the corresponding partition middle.

[0018] Step 6: Based on the partitioning results, assign all nodes and edges to corresponding distributed storage nodes to achieve balanced partitioning and storage.

[0019] The beneficial effects of the present invention are:

[0020] 1. By introducing representative subgraphs and a dual-scan strategy, this invention can complete the partitioning with only two disk scans. Compared with existing algorithms that require multiple full-graph traversals, this method significantly reduces I / O overhead and runtime, making it particularly suitable for processing ultra-large-scale graph data.

[0021] 2. The present invention uses random edge sampling combined with degree biased sampling (DBS) to construct representative subgraphs, which can complete the partitioning without loading the entire graph, effectively reducing memory requirements, adapting to memory-constrained scenarios, and extending to ultra-large-scale graph data that cannot be fully loaded.

[0022] 3. This invention significantly reduces the number of cross-partition edges (edge-cutting ratio) and improves load balancing between storage nodes through a gain-based partitioning algorithm. Experiments show that this invention outperforms current mainstream algorithms in partitioning quality and operational efficiency, with particularly significant advantages in low-memory scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is the overall process of an efficient financial knowledge graph division method in this embodiment. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below based on the accompanying drawings and preferred embodiments. The specific embodiments described herein are only used to explain the present invention but are not intended to limit the scope of protection of the present invention.

[0025] like Figure 1 As shown, an efficient method for dividing financial knowledge graphs includes the following steps:

[0026] Step 1: Initialize the sampling set E S and the vertex set V S ; The specific steps are as follows:

[0027] Step 1.1: From the edge sequence E seq Load the initial edge set and read E seq The first ρ edges in the middle, and each edge e=(u,v) is stored in the sampling set E in the format of (e,w,r,k) S , where w is the weight, r is the random value, and k is the key value (w, r, k are initially 0);

[0028] Step 1.2: Update the vertex set V S , for the two vertices u and v of edge e, update the information according to their degree values. The calculation formula is: V S =V S ∪(u,V S [u].d+1)∪(v,V S [v].d+1);

[0029] Step 1.3: Calculate E S The weight and key value of the middle edge, traversing E S For each edge e=(u,v), calculate its attributes according to the following rules r = random(0,1), k = r 1 / w .

[0030] Step 2: Dynamically update the sampling set E based on degree biased sampling (DBS) S ;

[0031] Step 2.1: From the edge sequence E seq Starting at position ρ+1+i·η, read the next η edges into the temporary set E′, where i=0,1,2,... represents the index of the current batch;

[0032] Step 2.2: Update the vertex set V according to each edge e = (u, v) in E′ S Vertex degree information, recalculate E S The key value k of each element in , and set the element with the smallest key value as the current minimum element T;

[0033] Step 2.3: Dynamically update the sampling set E according to the information of each edge e = (u, v) in E′ S ;

[0034] Step 2.3.1: Calculate the weight w, random value r, and key value k for each edge e = (u, v) in E′ using the following formula: r = random(0,1), k = r 1 / w ;

[0035] Step 2.3.2: Replacement sampling set E S Elements in: Decide whether to replace E based on the key value k of each edge e=(u,v) calculated in step 2.3.1 S If k>Tk, replace E with the current edge (e,w,r,key) S The element T with the smallest key value in the array is updated to E. S The new minimum key value element in, otherwise, continue to judge the next edge, always keeping E S The elements with large weight w are stored in .

[0036] Step 3: Repeat step 2 until the edge sequence E seq All edges in are processed completely, and the final ones retained in the sampling set E are used. S The edges in the graph represent the subgraph G S =(V S ,E S ).

[0037] Step 4: For the representative subgraph G S Apply the gain-based partitioning algorithm to divide it into k subgraphs, denoted as Divide each subgraph Save to the corresponding storage node.

[0038] Step 5: Process E seq For the unsampled vertices and edges in E, according to the vertex information of each unsampled edge, the edge is saved to the corresponding edge set. seq For each unsampled edge e=(u,v), if its vertex u or v is not in the vertex set V S In the example, it is assigned to a storage node i according to its locality. If vertices u and v belong to different partitions (i.e., i≠j, where u∈V i s and ) marks the edge as a "cross-partition edge (cut edge)" and saves it to the two partitioned edge sets ( and ); If vertices u and v belong to the same partition, store the edge in the edge set of the corresponding partition middle.

[0039] Step 6: Based on the partitioning results, assign all nodes and edges to corresponding distributed storage nodes to achieve balanced partitioning and storage.

[0040] Example 1:

[0041] A summary of the real and synthetic graphs used in the experiments is shown in Table 1. Duplicate edges, self-loops, symbols, and weights are removed. All real-world graphs are publicly available. All algorithms are implemented in C++ while simulating memory limitations. The KL partitioning algorithm is used to partition the sample graphs in all experiments. The evaluation metric used is edge cut rate. Measures the proportion of cut edges; Normalized maximum load (τ): measures the load balance between partitions; Running time (s): records the algorithm execution time.

[0042] Table 1: Data table used in the experiment

[0043]

[0044] The following heuristics are evaluated: balanced partitioning (B), hash partitioning (Hash), deterministic greedy (DG), linearly weighted deterministic greedy (LDG), exponentially weighted deterministic greedy (EDG), triangle-based partitioning (Tri, LTri, ETri), non-neighbor partitioning (NN) and FENNEL (FNL). For FENNEL, the parameter used throughout the experiment is set to γ ​​= 1.5, v=1.1.

[0045] This paper measures the performance of our algorithm against other stream partitioning algorithms on a set of real-world graphs. These experiments were run by partitioning the input graph on a set of k = 4 target partitions, with memory limits set to 30%, 50%, and 80%. The runtimes are reported in Tables 2, 3, and 4. The cut-edge fractions are reported in Tables 5 and 6. The normalized maximum loads are reported in Tables 7 and 8.

[0046] Table 2: Operation schedule when ρ = m × 30%, k = 4 (unit: seconds)

[0047]

[0048]

[0049] Table 3: Operation schedule when ρ = m × 50%, k = 4 (unit: seconds)

[0050]

[0051] Table 4: Operation schedule when ρ = m × 80%, k = 4 (unit: seconds)

[0052]

[0053]

[0054] As can be seen from Tables 2 to 4, our method demonstrates significant runtime advantages under memory constraints. When the memory limit is 30%, its runtime is only 26.5% of that of the FNL algorithm, and when the memory limit is 50%, its runtime is only 74.1% of that of FNL. In the case of larger memory (80%), existing algorithms such as FNL have slightly lower runtimes than our method, as they can load larger approximate graphs. However, overall, our method maintains high efficiency, especially in low-memory conditions, where it exhibits more stable performance by effectively avoiding extensive disk I / O operations.

[0055] In terms of edge cut rate, our method outperforms existing streaming partitioning algorithms across all memory configurations, particularly compared to the FNL algorithm. When the memory limit is 30%, the edge cut rate is reduced by 2.4% compared to FNL; when the memory limit is 50%, the edge cut rate is reduced by 6.7% compared to FNL; and when the memory limit is 80%, the edge cut rate is further reduced by 9.1%. These results demonstrate that our method effectively reduces the number of edges cut during partitioning, thereby improving partitioning quality.

[0056] In terms of load balancing, our method performs close to the theoretical optimality under all memory configurations, second only to the balanced partitioning algorithm, but significantly outperforming existing algorithms such as FNL. Specifically, compared to the FNL algorithm, its load balancing performance improves by 5.6%, 5.4%, and 5.4% when the memory limit is 30%, 50%, and 80%, respectively. This demonstrates that our method can effectively reduce load differences between partitions while maintaining partition quality, ensuring load balancing among partitions.

[0057] Furthermore, experimental results demonstrate that our method is less sensitive to memory constraints. In comparison, the FNL algorithm's runtime increased by 148.5% when the memory usage was reduced from 80% to 50%, and by 240.2% when the memory usage was further reduced to 30%. However, our proposed method performed reliably under the same conditions, increasing its runtime by only 12.4% when the memory usage was increased from 30% to 50%, and by only 10.5% when the memory usage was increased from 50% to 80%. This result further demonstrates that our method can operate efficiently in memory-constrained environments, significantly reducing the performance loss caused by insufficient memory.

[0058] Table 5: Edge cutting rate table when k=4

[0059]

[0060]

[0061] Table 6: Edge cutting rate when k = 4, ρ = 0.3, 0.5, 0.8

[0062]

[0063] Table 7: τ comparison table when k = 4

[0064]

[0065]

[0066] Table 8: Comparison of τ when k = 4, ρ = 0.3, 0.5, 0.8

[0067]

[0068] In summary, experimental results demonstrate that the proposed method exhibits superior performance in terms of edge-cutting rate, load balancing, and runtime. Particularly under memory-constrained conditions, this method effectively improves efficiency by avoiding extensive disk I / O operations, while achieving good load balancing while maintaining partitioning quality. Compared to existing streaming partitioning algorithms, this method demonstrates significant practical value and performance advantages for large-scale graph partitioning tasks.

[0069] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been described in detail with reference to the aforementioned examples, researchers in this field may still make minor modifications to the technical solutions described in the aforementioned examples, or replace some of the technical features therein with equivalents. Any modifications, equivalent variations, and alterations made within the spirit and principles of the invention shall fall within the scope of the technical solution of the present invention.

Claims

1. An efficient method for dividing financial knowledge graphs, characterized by: The steps include: Step 1: Initialize the sampling set E S and the vertex set V S ; Step 2: Dynamically update the sampling set E based on degree biased sampling (DBS) S ; Step 3: Repeat step 2 until the edge sequence E seq All edges in are processed completely, and the final ones retained in the sampling set E are used. S The edges in the graph represent the subgraph G S =(V S ,E S ); Step 4: For the representative subgraph G S Apply the gain-based partitioning algorithm to divide it into k subgraphs, denoted as Divide each subgraph Save to the corresponding storage node; Step 5: Process E seq For the unsampled vertices and edges in the , according to the vertex information of each unsampled edge, save the edge to the corresponding edge set; Step 6: Based on the partitioning results, assign all nodes and edges to corresponding distributed storage nodes to achieve balanced partitioning and storage.

2. An efficient financial knowledge graph partitioning method according to claim 1, characterized in that: The specific method in step 1 is: Step 1.1: From the edge sequence E seq Load the initial edge set and read E seq The first ρ edges in the middle, and each edge e=(u,v) is stored in the sampling set E in the format of (e,w,r,k) S , where w is the weight, r is the random value, and k is the key value; w, r, k are initially 0; Step 1.2: Update the vertex set V S , for the two vertices u and v of edge e, update the information according to their degree values. The calculation formula is: V S =V S ∪(u,V S [u].d+1)∪(v,V S [v].d+1); Step 1.3: Calculate E S The weight and key value of the middle edge, traversing E S For each edge e=(u,v), calculate its attributes according to the following rules r = random(0,1), k = r 1 / w .

3. An efficient financial knowledge graph partitioning method according to claim 2, characterized in that: In step 1.3, the sample set E is calculated S The weight w, random value r and key value k of each edge e=(u,v) in each edge are used to update the sampling set E in the next step S .

4. An efficient financial knowledge graph partitioning method according to claim 1, characterized in that: The specific method in step 2 is: Step 2.1: From the edge sequence E seq Starting from position ρ+1+i·η, read the next η edges into the temporary set E′, where i=0,1,2,… represents the index of the current batch; Step 2.2: Update the vertex set V according to each edge e=(u,v) in E′ S Vertex degree information, recalculate E S The key value k of each element in , and set the element with the smallest key value as the current minimum element T; Step 2.3: Dynamically update the sampling set E according to the information of each edge e = (u, v) in E′ S .

5. An efficient financial knowledge graph partitioning method according to claim 4, characterized in that: In step 2.3, the sampling set E S To update, the specific steps are as follows: Step 2.3.1: Calculate the weight w, random value r, and key value k for each edge e = (u, v) in E′ using the following formula: r = random(0,1), k = r 1 / w ; Step 2.3.2: Replacement sampling set E S Edges in: Decide whether to replace E based on the key value k of each edge e=(u,v) calculated in step 2.3.1 S If k>Tk, replace E with the current edge (e,w,r,key) S The edge T with the smallest key value in the middle, and update T to E S The new minimum key value element in, otherwise, continue to judge the next edge, always keeping E S The edges with large weight w are stored in .

6. An efficient financial knowledge graph partitioning method according to claim 1, characterized in that: In step 5, E seq The specific strategy for unsampled edges and vertices is: for E seq For each unsampled edge e=(u,v), if its vertex u or v is not in the vertex set V S In the example, it is assigned to a storage node i according to its locality. If vertices u and v belong to different partitions (i.e., i≠j, where u∈V i s and ) marks the edge as a "cut edge" and saves it to the two partitioned edge sets ( and ); If vertices u and v belong to the same partition, store the edge in the edge set of the corresponding partition middle.

Images