Incremental graph mining method and system based on edge update driving in financial transaction dynamic graph

Abstract

The application discloses a kind of increment graph mining method and system based on edge update driving in financial transaction dynamic graph, belong to graph mining technical field.The method includes: constructing financial transaction graph;According to a plurality of data operations facing the financial transaction graph, generate the work queue containing a plurality of update tasks;Multiple computing kernels are parallelly taken update task from work queue;With the edge in data operation as starting point, carry out graph mining, and combine the financial transaction graph snapshot before updating, obtain the difference data corresponding to each update task;Aggregate the difference data, obtain graph mining result.The application can optimize non-uniform data distribution scene, improve throughput, video memory utilization and system stability.

Description

Technical Field

[0001] This invention belongs to the field of graph mining technology, specifically relating to an incremental graph mining method and system based on edge update-driven approach in dynamic graphs of financial transactions. Background Technology

[0002] Graphs are a data representation consisting of vertices and edges, capable of efficiently describing entities and their relationships in the real world. Algorithms that match specified patterns on graph data are called graph mining algorithms. In recent years, they have demonstrated significant importance in applications such as financial transactions, protein property prediction, network alignment, anomaly detection, entity disambiguation, community detection, role discovery, and large model retrieval enhancement generation. However, their execution efficiency on large-scale graphs still needs improvement. Furthermore, graph data is subject to frequent changes, so graph data systems need to quickly respond to modifications to the graph data while efficiently updating the results of graph mining algorithms for downstream applications to access.

[0003] In financial transaction scenarios, the method of constructing graphs based on financial transaction data involves treating the entities participating in financial activities (such as accounts, companies, and consumers) as vertices in the graph data, and the interactions between these entities (such as purchasing goods and transferring funds) as edges between the vertices. For example, when account A transfers funds to account B, this is recorded in the graph data as adding an edge labeled "transfer" between account vertex A and account vertex B. In financial transaction scenarios, certain specific subgraph patterns in the aforementioned graph data have practical significance and require rapid identification and location by the system. For example, when the transfer connections between multiple accounts form a loop, it indicates that these accounts may be involved in money laundering or other fraudulent activities.

[0004] Currently, existing graph data representation technologies in this scenario have certain shortcomings in terms of algorithms and efficiency. Taking the aforementioned fraudulent behavior as an example, traditional graph mining relies on static graph snapshots or periodic reconstructions, which cannot capture dynamic relationship changes in the transaction timeline. For instance, fraudsters may suddenly activate dormant accounts and mimic the transit paths of normal users, causing severe lag in matching based on historical pattern libraries. Furthermore, the evolution rate of fraudulent behavior far exceeds the iteration speed of rules, and existing methods lack adaptive mechanisms to discover novel topological structures that have never appeared in historical samples. Simultaneously, when the graph data scale reaches hundreds of millions of edges or more, the query overhead for graph data storage is extremely high, and the edge labels of multi-source heterogeneous data are complex and updated rapidly. Existing technologies generate a large number of access requests to graph data storage, forming a processing latency bottleneck. Applying this invention can accelerate the response speed to dynamic updates of graph data, and simultaneously output the updated set of graph mining algorithm results after each round of edge updates, thereby keeping abreast of dynamic relationship changes in the graph data more promptly.

[0005] To meet the aforementioned dynamic graph processing requirements, existing technologies have attempted to optimize from different angles, mainly including bit vector-based compressed storage, adaptive radix tree indexing, and CPU-side multi-threaded dynamic graph structures.

[0006] 1. Neighbor list representation based on bit vectors.

[0007] To accelerate set intersection operations, Han et al. (HAN S, ZOU L, YU J X. Speeding Up SetIntersections in Graph Algorithms using SIMD Instructions[C] / / Proceedings of the 2018 International Conference on Management of Data. ACM, 2018:1587-1602.) proposed a neighbor list representation method based on bit vectors. This technique encodes the neighbor list of a vertex into a binary bit string, where the w-th bit is 1 to indicate the existence of a connection and 0 to indicate the absence of a connection. To address the space waste caused by sparsity, this scheme divides the bit string into fixed-length data blocks and stores them using position indices. However, this scheme is mainly designed for static graphs and has limited support for dynamic updates. Although the insertion operation seems simple, adjusting the position index pointer is required when inserting a new data block, resulting in a high cost for maintaining order. Furthermore, its space efficiency is highly dependent on the degree of clustering of vertex IDs. In dynamic scenarios with uneven ID distribution, it is easy to cause space waste (in the worst case, only one bit is stored for each data block), and it is difficult to directly migrate to GPU architecture for execution.

[0008] 2. Adaptive Radix Tree (ART).

[0009] The Adaptive Radix Tree (ART) proposed by Viktor Leis et al. (LEIS V, KEMPER A, NEUMANN T. The adaptive radix tree: ARTful indexing for main-memory databases[C] / / ICDE. IEEE Computer Society,2013: 38-49.) reduces memory usage while maintaining high-speed query capabilities by dynamically adjusting the internal structure of nodes (e.g., Node 4, Node 16, Node 48, Node 256). However, ART was originally designed for in-memory databases, and its nodes undergo frequent structural upgrades as data is inserted (e.g., from Node 4 to Node 16). This frequent dynamic memory allocation and structural transformation is extremely detrimental to GPUs. The SIMT architecture of GPUs requires threads within a warp to execute synchronously. The structural changes in ART can lead to severe warp fragmentation, disrupting the GPU computational pipeline, and its concurrency control is difficult to achieve efficiently in a GPU environment.

[0010] 3. CPU-side dynamic graph storage structure (Spruce).

[0011] Spruce, proposed by Shi et al. (Shi J, Wang B, Xu Y. Spruce: a Fast yet Space-saving Structure for Dynamic Graph Storage[J]. Proceedings of the ACM on Management of Data, 2024, 2(1):1-26.), is a technique that utilizes CPU multi-threading to process graph updates. It combines multi-level vertex indexing and hybrid edge storage (insertion buffer + ordered array), and is equipped with lightweight concurrency control mechanisms (such as improved ROWEX, optimistic locking, etc.). However, Spruce is designed entirely based on CPU architecture. Although it handles concurrency control, it does not consider the characteristics of GPU hardware. Directly porting the CPU-side locking mechanism and complex data structures to the GPU will result in a sharp performance drop due to a large number of atomic operations and thread blocking. In addition, although Spruce considers dynamic updates, it lacks targeted optimizations when facing GPU-specific memory merging access and irregular memory access patterns.

[0012] Furthermore, although existing technologies have laid the foundation for dynamic graph management, there are still significant technological gaps and deficiencies in utilizing GPUs for acceleration to achieve dynamic graph updates and incremental updates of graph mining results.

[0013] 1) Existing technologies have not fully explored the design and optimization techniques for GPU-accelerated dynamic graph mining. Numerous works have emerged in this field utilizing GPU hardware to accelerate graph analysis and data processing. GPUs have experienced significant improvements in computational power and accessibility, making them a viable option for addressing the challenges of dynamic graph data structures. Modern GPUs contain thousands of computational cores, which, while individually weak, work collaboratively through a Single Instruction Multithreaded (SIMT) architecture. This parallelism is particularly important in graph algorithms, as many algorithms (such as labeling, filtering, and intersection operations), while simple, require independent execution on a large number of vertices or edges. While GPU solutions proposed in other fields have significantly improved the efficiency of large-scale graph processing through deep optimization of memory access patterns and computational parallelism, existing technologies still suffer from high concurrency control costs and low storage resource utilization efficiency in terms of dynamic graph data structures and updating graph mining algorithm results. Designing fully optimized dynamic graph data structures on GPUs requires a thorough consideration of the inherent characteristics of GPUs. Simply porting CPU-based systems to GPU platforms will not achieve optimal performance; specialized optimization strategies for the GPU architecture must be employed. Specifically, conditional branching statements can lead to thread bundle fragmentation. Since all threads within the same bundle must execute the same instructions synchronously, some threads will be forced to remain idle. Furthermore, unevenly distributed graph structures further exacerbate the memory locality problem. By rationally utilizing shared memory—a programmable, high-speed on-chip storage resource—it can be configured as a user-defined cache, effectively reducing the number of off-chip memory accesses. Other key optimization dimensions include: implementing memory access merging to improve bandwidth utilization, designing efficient memory overflow handling mechanisms, and achieving workload balancing through dynamic scheduling. These GPU-specific optimizations collectively constitute the fundamental design principles of high-performance dynamic graph data structures. This invention fully incorporates the design ideas of existing technologies and combines them with GPU hardware characteristics for system design and optimization, aiming to improve the time and space efficiency of dynamic graph updates and graph mining algorithm result updates on the GPU.

[0014] 2) Optimizing the performance of applications on GPUs requires a deep understanding of the memory access characteristics of the workload. The operations and graph algorithms required in dynamic graph data structures are often accompanied by unpredictable memory allocation and random access to adjacency lists. For example, continuous edge insertion may trigger data structure expansion, thus interrupting GPU processing; when algorithms enumerate vertices by adjacent vertices, highly irregular memory access patterns arise. Due to the inherent low spatial and temporal locality of the workload, predicting the next data access location becomes extremely difficult. This forces many existing systems to pre-allocate large amounts of memory and organize them into fixed-size blocks or pages. Researchers have also proposed various adjacency list storage strategies to accelerate access to adjacent vertices. Notably, optimization techniques for improving locality in static graphs face significant challenges in dynamic graph scenarios. Continuous graph updates require pausing graph update operations during reordering, newly inserted edges disrupt the original order, and frequent reordering consumes substantial computational resources. These limitations necessitate the development of innovative optimization schemes specifically tailored to the characteristics of dynamic graphs.

[0015] 3) In existing technologies, GPU-based techniques do not consider special processing for the neighbor lists of vertices with high-order vertices, while some non-GPU-based techniques mention using storage methods more suitable for large-scale dynamic data for the extremely large neighbor lists of a few vertices with high-order vertices. These two approaches have not been fully explored and integrated. In the storage and real-time updating scenarios of dynamic graphs, real-world graph data typically exhibits a significant power-law distribution: taking a typical social network graph as an example, over 90% of vertices have a degree less than 20, while less than 2% of vertices have degrees reaching thousands or even higher (e.g., the maximum degree in the Twitter dataset reaches approximately 2.997 × 10⁻⁶). 6 The extreme skewness of this degree distribution means that the size that the vertex neighbor list needs to accommodate varies greatly globally. Therefore, the data structure selection of the neighbor list should not be unrelated to the vertex degree. The dynamic graph storage scheme under the GPU acceleration route has not yet absorbed the CPU side's understanding that "degree skewness → the neighbor list structure should not be a one-size-fits-all approach". It still maps the adjacent data of all vertices to the same GPU-friendly scalable container. On the other hand, the CPU side's designs that clearly realize that vertices with high degree need to be treated differently have not been extended into a complete scheme that can work in conjunction with the GPU side's storage management and parallel update mechanism. There is a lack of systematic integration between the two, with vertex degree driving the heterogeneity of the neighbor list structure as a bridge. As a result, the combination optimization space that simultaneously satisfies the three constraints of (1) large-scale dynamic scaling of the neighbor list of high-degree vertices, (2) space compactness of the low-degree vertex side, and (3) regularity and load balancing of the GPU execution model has not been fully explored and utilized.

[0016] In summary, current technologies have not yet provided a dynamic graph data structure and processing method that can effectively combine the advantages of CPU-GPU heterogeneous computing platforms. How to design a data structure on a GPU that can adapt to high-frequency dynamic updates, perform differentiated storage for vertices with high height, and avoid thread bundle splitting and memory access bottlenecks is a pressing technical problem that needs to be solved. Summary of the Invention

[0017] This invention discloses an incremental graph mining method and system based on edge update-driven approach in dynamic graphs of financial transactions. It leverages the characteristics of GPU hardware to decompose dynamic graph mining tasks, designs data structures and computing kernels adapted to GPUs, optimizes non-uniform data distribution scenarios, and improves throughput, memory utilization, and system stability.

[0018] To achieve the above objectives, the technical solution of the present invention includes the following:

[0019] An incremental graph mining method based on edge update-driven approach in a dynamic graph of financial transactions, the method comprising: Construct a financial transaction graph, where edges in the graph represent the interaction behavior of vertices. The types of vertices include accounts, companies, and consumers, and the interaction behavior includes purchasing goods and transferring funds. Based on several data operations oriented towards the financial transaction graph, a work queue containing several update tasks is generated; wherein, the data operations include edge insertion or deletion operations; Multiple computing cores retrieve update tasks from the work queue in parallel; Graph mining is performed starting from the edges in the data operations, and combined with the snapshot of the financial transaction graph before the update, the difference data corresponding to each update task is obtained. The difference data is aggregated to obtain the graph mining results.

[0020] Furthermore, the vertex index of the vertex includes a hash layer and a vector index layer; the vertex index is used to locate the neighbor list of the vertex in the data layer based on the vertex identifier; the vertex identifier's prefix... The index of the vertex identifier is maintained by a hash layer. Each byte is indexed by a vector index layer. , It is a positive integer, and , The length of the vertex identifier; Furthermore, the vector index layer is an L-layer index tree structure, where each layer of the index tree indexes one byte; wherein, the vector index layer includes full mapping nodes and / or compressed mapping nodes; the full mapping node contains a length of 2... An array of pointers to a power of 2, wherein the array of pointers to the full-mapped node points to a node in the next level of the index tree or a list of neighbors in the data layer; the compressed mapping node contains a 2 The power of a bit vector and a length of 2. An array of pointers to the power of 1, wherein the array of pointers to the compressed mapping node points to the node of the next level of the index tree or the neighbor list of the data layer, and the bit vector is used to identify the existence status of the node of the next level or the neighbor list of the data layer.

[0021] Furthermore, when the degree of a vertex is greater than a set threshold, the vertex index of the vertex also includes a fast channel for the height-number vertex; wherein the fast channel for the height-number vertex is implemented based on a Bloom filter and maintains a list of pointers to the neighbor list of the vertex.

[0022] Furthermore, in the case where the data operation is an edge insertion operation, the step of generating a work queue containing several update tasks based on several data operations oriented towards the financial transaction graph includes: Obtain the vertex identifier corresponding to the edge to be inserted, perform multi-level positioning operations based on the vertex identifier, and find the insertion buffer to be written; Using the previous vertex identifier The byte hash value locates the storage address of the first-level node in the vector index layer; based on the last vertex identifier... Calculate the bit vector position corresponding to the vertex identifier byte by byte, layer by layer; If the current node is a compressed mapping node and the corresponding bit vector position is 1, or if the current node is a full mapping node and the corresponding position index pointer is not null, then jump to the next level along the pointer; if the jump condition is not met, then set the corresponding bit vector position of the compressed mapping node to 1, and initialize the next level node and the corresponding pointer. Repeat the hierarchical jump steps and determine the target insertion buffer by using the last node of the vector index layer; Read the tail counter of the target insertion buffer to obtain the write position: if the tail counter has not reached the upper limit of the insertion buffer capacity, write the edge data directly, generate the corresponding update task, and update the element counter and tail counter; if the tail counter has reached the upper limit of the insertion buffer capacity, append the edge data to the sorted data area and generate the corresponding update task.

[0023] Furthermore, in the case where the data operation is an edge deletion operation, the step of generating a work queue containing several update tasks based on several data operations oriented towards the financial transaction graph includes: Locate the insertion buffer corresponding to the target vertex, and scan the insertion buffer to find the edge to be deleted; If the edge to be deleted is not found in the insertion buffer, the process jumps to the sorted data area and uses a binary search to locate the edge to be deleted. After modifying the data of the edge to be deleted to a preset special deletion marker value, a corresponding update task is generated.

[0024] Furthermore, the data operation also includes an edge query operation; wherein the query operation includes: Perform multi-level indexing and location operations based on vertex identifiers to find the neighbor list corresponding to the target vertex; If the neighbor list of the target vertex is successfully located, the corresponding edge query operation is performed based on the query requirements: if the query requirement is a neighborhood scan requirement, the insertion buffer and sorted data area are scanned sequentially to obtain the complete neighbor list of the target vertex; if the query requirement is an edge existence check requirement, the insertion buffer is scanned and the sorted data area is searched to locate the target vertex and determine whether the corresponding edge exists.

[0025] Furthermore, graph mining is performed starting from the edges in data operations, including: Initialize the matching subgraph based on the edges in the data operations; Select an unmatched vertex from the financial transaction graph that is adjacent to the matching subgraph and satisfies the query pattern constraints; Add the unmatched vertex and the edges between the unmatched vertex and vertices in the matched subgraph to the matched subgraph to obtain the extended subgraph; Within the extended subgraph, undo the edges in the data operation to obtain the subgraph before the update; The algorithm checks the subgraph before and after the update to determine if it can be further expanded. If it can be further expanded, it verifies whether the subgraph's connectivity, vertex and edge labels, and topology satisfy the query pattern constraints. If the subgraph before the update satisfies the constraints, it outputs the removed matching results; if the subgraph after the update satisfies the constraints, it outputs the newly added matching results. When the subgraph before update or the subgraph after expansion has expansion potential, the step of selecting an unmatched vertex from the financial transaction graph that is adjacent to the matching subgraph and satisfies the query pattern constraint is re-executed based on the expanded subgraph. When neither the subgraph before the update nor the subgraph after the expansion has expansion potential, select another unmatched vertex from the financial transaction graph that is adjacent to the matching subgraph and satisfies the query pattern constraint, and re-execute the step of adding the unmatched vertex and the edge between the unmatched vertex and the vertex in the matching subgraph to the matching subgraph to obtain the expanded subgraph.

[0026] An incremental graph mining system based on edge update-driven approach in a dynamic graph of financial transactions, the system comprising: The graph construction module is used to construct a financial transaction graph, in which the edges represent the interaction behavior of vertices. The types of vertices include accounts, companies, and consumers, and the interaction behavior includes purchasing goods and transferring funds. The work queue generation module is used to generate a work queue containing several update tasks based on several data operations oriented towards the financial transaction graph; wherein, the data operations include edge insertion or deletion operations; The graph mining module is used by multiple computing kernels to retrieve update tasks from the work queue in parallel; it performs graph mining starting from the edges in the data operations, and combines the snapshot of the financial transaction graph before the update to obtain the difference data corresponding to each update task; it aggregates the difference data to obtain the graph mining results.

[0027] A computer device includes: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements the edge-update-driven incremental graph mining method in the dynamic graph of financial transactions as described above.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects.

[0029] 1. This invention fully utilizes the characteristics and advantages of GPU hardware, dividing the dynamic graph mining task into two stages: responding to dynamic graph updates and searching for incremental graph mining results, and providing technical solutions for each. By designing a hardware-friendly dynamic graph data structure and GPU-based graph mining computation kernel processing logic, it achieves higher computational throughput and GPU storage utilization efficiency.

[0030] 2. We conducted an in-depth analysis of the algorithmic behavior of dynamic graph mining on GPUs and the data distribution characteristics of the dynamic graphs themselves. We made special treatments for uneven data distribution, which made the system performance more stable and able to adapt to different data distributions and workloads. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the dynamic graph data structure in this invention.

[0032] Figure 2 This is a flowchart illustrating the execution of the dynamic graph mining algorithm on a GPU. Detailed Implementation

[0033] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the objectives, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0034] (a) Dynamic graph data structure section.

[0035] Taking financial transaction scenarios as an example, the construction of dynamic graph data begins with the real-time collection and offline batch synchronization of multi-source heterogeneous data. Collection sources include transaction logs, user registration and login behavior, device fingerprints, IP address and geographic location, and merchant classification information. The collected raw data undergoes preprocessing: first, missing values ​​are filtered using thresholds or filled with rules (e.g., assigning default geocoding to unknown IPs), and timestamps are aligned and sliced ​​using sliding windows to eliminate out-of-order effects; second, entity resolution technology is used to resolve identity conflicts and unify the referencing of user IDs, bank card numbers, device IDs, etc., scattered across different systems, mapping multiple internal IDs of the same person across different business lines to a unique virtual node ID; finally, continuous attributes such as transaction amount and frequency are discretized, and discrete enumeration fields are standardized to ensure semantic consistency for subsequent graph queries. During the dynamic graph construction phase, adding new edges (transaction relationships) is sequentially applied according to fixed time windows (e.g., seconds, minutes, or days). In this dynamic graph structure, the definition of nodes follows the abstract layering of business semantics: subject nodes include personal accounts, corporate accounts, bank card numbers, and device fingerprints, while object nodes cover IP addresses, Wi-Fi MAC addresses, merchant numbers, and geographic location grids; each dynamic edge represents a specific financial interaction event, and its definition must simultaneously cover the topological direction (funds flowing out to funds flowing in), interaction type (transfer, payment, binding, login), and highly time-sensitive attributes (transaction timestamp, amount, cumulative frequency value). The attribute fields on the edge are updated in real time or expire with each window slide.

[0036] Figure 1 This invention demonstrates the dynamic graph data structure design. The hash layer, vector index layer, and height-based vertex fast path together constitute the vertex index, which quickly locates the neighbor list of a vertex within the data layer based on its vertex identifier. The data layer stores the neighbor list, including three different storage methods. The data structure components shown in the diagram will be described in turn below.

[0037] The vertex identifier in this invention is used to map a vertex to its associated list of neighbors. For ease of description, this section uses a 4-byte unsigned integer vertex identifier as an example. Figure 1 As shown, the vector index layer is an index tree structure, with each level of the tree indexing one byte. Let the number of levels be L. L is a configurable parameter. For a 4-byte unsigned integer vertex identifier, choosing L=2 yields the best performance. In this case, the last 2 bytes of the vertex identifier are indexed by the vector index layer, while the first 2 bytes are indexed by the hash layer.

[0038] Hash Layer: The first two bytes of the vertex identifier are mapped to different cells in the hash table through a hash function. These cells store the addresses used to point to child nodes in the first layer of the vector index layer. That is, all vertices with the same 2-byte prefix are mapped to the same vector index layer. This design reduces the impact of uneven distribution of dynamic graph data on load balancing through prefix clustering and random mapping of the hash function. The choice of hash table technology in this invention has three advantages: First, vertex identifiers with the same 2-byte prefix will share the same hash address, which can significantly save storage space; Second, lookup and modification operations in the hash table can be completed in O(1) time complexity, which is better than the O(log(n)) complexity of structures such as B-trees, and is especially suitable for high-concurrency query scenarios; Third, hash tables are a very commonly used data structure, and there are mature design schemes and library functions available on both CPU and GPU sides.

[0039] Vector Index Layer: The last 2 bytes of the vertex identifier are maintained in the vector index layer by a tree structure similar to an adaptive radix tree. There are two different types of nodes in the vector index layer of this invention: (1) Full mapping node, which contains a pointer array of length 2^8. The array is indexed by a complete 1-byte value of 8 bits. Each pointer points to a node in the next layer or a corresponding block in the data layer. In the initial state, the pointer array is empty, indicating that no vertex with the corresponding vertex identifier has been inserted. The first layer nodes of the vector index layer are full mapping nodes by default. This design only requires less pre-allocation of space, but can avoid a large number of dynamic adjustment operations of nodes; (2) Compact mapping node, which contains a bit vector of length 2^8 bits and a pointer array of length 2^7 bits. The bit vector is used to identify the existence status of the 1-byte value, and the pointer array points to a node in the next layer or a corresponding block in the data layer. The bit vector uses a 1-bit marking method: 1 indicates that the value exists, and 0 indicates that it does not exist. This design, through bit vectors, not only enables rapid location of individual data blocks using vertex identifiers, but also accelerates vertex range scanning operations in dynamic graph processing, where deleting a vertex requires quickly locating and removing all its adjacent edges. It's important to note that the transformation of nodes in the vector index layer—that is, the transformation from compressed mapped nodes to full mapped nodes—only occurs during the structural adjustment phases of the two update batches.

[0040] Fast path for height-number vertices: In real graphs, the number of height-number vertices is much smaller than the total number of vertices. Moreover, in this invention, an adaptive storage method is adopted for height-number vertices. Therefore, it is necessary to build a lighter index for height-number vertices separately, rather than inserting them into the vertex index composed of the hash layer and the vector index layer. This invention uses a Bloom filter to implement a fast path for height-number vertices, while maintaining a list of pointers to the neighbor list of height-number vertices. The pointers are sorted according to the size order of the corresponding vertex identifiers. There are two advantages to using a Bloom filter: (1) Since the number of height-number vertices is very small, there are very few edges to be inserted in the Bloom filter. According to the formula commonly used when constructing a Bloom filter to calculate the number of bits m required by the Bloom filter based on the expected error rate p and the number of elements to be inserted n, the smaller the number of elements to be inserted n, the fewer bits m are required while maintaining the same false alarm rate p. In other words, the fast path for height-number vertices in this invention only needs to maintain a very small Bloom filter to meet the requirements. (2) Although the Bloom filter has errors, the only error is when an element that has not been inserted is incorrectly judged as having been inserted. This can be eliminated by performing a binary search in the list of pointers to the neighbor list of heights, since the number of height vertices is very small and the time complexity of binary search of O(log(n)) is acceptable.

[0041] This invention maintains a dynamic graph structure consistent with the one shown in the figure for each vertex whose number of neighbors exceeds an empirical threshold. This is because vertices with high height are more likely to carry more edge update operations in real-world applications, which would lead to excessive concurrency conflicts in the buffer insertion area in the original design, and it might be filled up quickly. This design can be seen as logically dividing the high-height vertices into multiple vertices and maintaining them using the dynamic graph structure in this invention. At this time, the user can choose the data representation method of the edge storage part according to the graph algorithm load to be carried. For scenarios that require scanning a large number of neighbor lists, the first k-1 bytes of the k-byte vertex identifier can be used as the key value maintained in the original vertex index part, that is, the neighbor vertex identifiers of the high-height vertices are clustered according to the first k-1 bytes, and the maintenance of neighbor vertices in each cluster still uses the original combination of insertion buffer and sorted data area; for scenarios that require a large number of edge existence checks and edge updates, another option is to represent the neighbor list of the high-height vertices according to the binary bit string representation method of Han et al., and use the block index value after the binary bit string is divided as the key value maintained in the vertex index part.

[0042] Insertion Buffer: Used for temporary storage of edge data. The insertion buffer consists of the following parts: a header, an element counter, a tail counter, a pointer to the sorted data area, and an array for storing edge data. The header mainly contains lock-related information. The element counter maintains the actual number of elements contained in the insertion buffer, i.e., the size of the neighbor list. If the neighbor list of a vertex is less than or equal to the maximum capacity of the insertion buffer (generally set to 64 in this invention), all edges of that vertex are stored in the insertion buffer; otherwise, the sorted data area is used to store more edges. The tail counter is used to quickly determine the insertion position. Newly inserted edges are written first according to the index position in the tail counter. When the tail counter reaches the maximum capacity of the insertion buffer, data is written to the tail of the sorted data area, but the insertion is performed in order. The contents of the insertion buffer and the sorted data area are then merged later. Edge data is not required to be ordered in the insertion buffer; it is arranged in the insertion order in the edge data array.

[0043] The sorted data area is an extension of the insertion buffer, used to transfer data from the insertion buffer when it overflows. The sorted data area includes a header and an array of neighbor identifiers. The header field records the capacity of the data area and the number of deleted edges. It's important to note that adjacent edges stored in the sorted data area are arranged consecutively in ascending order of vertex identifiers. When performing an edge deletion operation, this invention only marks the corresponding edge as invalid, rather than physically removing it. When the insertion buffer is full, its edges are transferred to the sorted data area, triggering merge sort to merge the newly added edges with the existing ordered sequence.

[0044] The data update operations supported in this invention include the insertion and deletion of vertices and edges, where the insertion and deletion of vertices can be implemented through the insertion and deletion of edges. The implementation logic of edge insertion, deletion, and query operations is described below.

[0045] The insertion operation first requires locating the insertion buffer based on the vertex identifier, and then modifying the insertion buffer. Specifically, the process involves first using the hash value of the first two bytes of the vertex identifier to locate the storage address of the first-level node in the vector index layer. Then, the bit vector position corresponding to the vertex identifier is calculated layer by layer using the subsequent bytes of the vertex identifier. The algorithm behavior is determined based on the value at the corresponding position: if the node is a compressed mapping and the corresponding bit vector position is 1, or if the node is a full mapping and the index pointer at the corresponding position is not null, then the pointer jumps directly to the next level; otherwise, the value at the corresponding position of the compressed mapping node's bit vector is modified to 1, and the nodes and their pointers in the next level are initialized. This process is repeated sequentially through the compressed index layer to locate the data layer, and finally, the insertion buffer corresponding to the vertex to be inserted is determined through the last-level node of the vector index layer. When modifying the insertion buffer, the tail counter is first read to obtain the write position. If the tail counter has not reached the upper limit of the insertion buffer capacity, the data can be written directly, and the element counter and tail counter are updated. If the tail counter reaches the upper limit of the insertion buffer, the data is appended to the sorted data area. This invention achieves fast positioning through multi-level bit vectors, with the operation time complexity of each positioning layer approaching O(1). The insertion operation requires synchronous updating of the index structure, and its execution process is similar to that of the positioning operation. The difference is that the corresponding bit in the bit vector is set layer by layer according to the vertex identifier. If a certain layer block does not exist, a new block is allocated and the pointer is written into the bit vector of the upper layer block. For example, when the vector index layer is not hit, a new vector index layer node is allocated, and when the data layer is not hit, a new insertion buffer is allocated.

[0046] The edge deletion operation requires two steps: locating the vertex and marking the deletion. First, the insertion buffer containing the vertex is located, and a scan is performed within it to find the edge to be deleted. If no edge is found, the process jumps to the sorted data area, where a binary search is performed to locate the edge to be deleted. This invention uses a classic lazy deletion strategy, not immediately reclaiming the deleted space during the deletion operation, but instead modifying its value to a special deletion marker. The deletion operation may cause a vertex's neighbor list to become empty. In this case, the corresponding vertex identifier needs to be recorded in a separate list for later use when adjusting the data structure.

[0047] During the location process, if the query operation successfully finds the neighbor list corresponding to the vertex identifier, it either scans the insertion buffer and sorted data area sequentially to obtain the complete neighbor list (neighborhood scan), or performs a scan or binary search in the insertion buffer and sorted data area respectively to find the target vertex (edge ​​existence check). The location process of the query operation may end prematurely because nodes in the insertion buffer and vector index layer without inserted edges may not exist. If a null pointer or the corresponding bit vector position is 0 is found during the location process, the operation can exit early and return the result that the queried element does not exist.

[0048] In the insertion and deletion process of this invention, there are several possible data structure adjustment operations that may be triggered: (1) When inserting an edge, it is necessary to initialize the vector index layer node, the insertion buffer, and the sorted data area. (2) When inserting an edge, if the space of the insertion buffer is exhausted, it is necessary to move the data to the sorted data area or reuse the deleted space. (3) When deleting an edge, if the deleted edge is the last edge that caused the vertex, the corresponding insertion buffer and sorted data area should be reclaimed, and the value of the corresponding bit in the bit vector of the compressed mapping node in the vector index area should be recursively modified. However, in order to avoid the impact of frequent data structure adjustments on the overall throughput during the update process, this paper designs an optimization strategy to delay the timing of dynamic adjustments. For operations (2) and (3) involving data movement and garbage collection, this invention delays them and executes them between two update batches. The vertex identifiers that have been deleted and whose space has been reclaimed are recorded separately, while the case of insertion buffer overflow requires traversing the vertex identifiers inserted in the previous batch to search. This dynamic adjustment process can overlap with the graph algorithm calculation on the GPU side, thereby reducing the impact on performance.

[0049] (ii) Dynamic graph mining algorithm.

[0050] This invention also includes an optimized scheme for dynamic graph mining algorithms on GPUs for large-scale graphs. The execution flow of the dynamic graph mining algorithm on the GPU in this invention is as follows: Figure 2 As shown, it includes the following steps: (1) Updates are received in batches from one or more data sources and are received by a CPU host as an entry node. This entry node assigns timestamps to the updates in ascending order and then applies the updates to a dynamic graph data structure that stores the timestamps. Before the update, the host preprocesses the update batch to remove pairs of insertions and deletions of the same edge. After the update, the host also filters the update batch to remove failed updates (insertions of existing edges and deletions of non-existent edges). The data storage used here is the aforementioned dynamic graph data structure. In this step, the edge insertion operation of the dynamic graph data structure needs to be performed. In subsequent steps, the edge data lookup operation of the data structure needs to be performed, that is, the neighbor list of the vertex is found based on the vertex identifier and traversed.

[0051] (2) The system inserts these updates into a set of work queues according to random allocation.

[0052] (3) After the computational kernel retrieves a task from the work queue, it executes the specific dynamic graph mining algorithm. At this time, each computational kernel executes a search method starting from the updated edge. This method operates on each update in the work queue and uses graph snapshots to calculate the differences, which are then sent to a result collection module. These differences are processed through aggregation operations to generate real-time graph mining results under an update batch. For financial trading scenarios, these results include entities represented by vertices and a set of behaviors performed by these entities corresponding to the edges in the graph matching results.

[0053] (3.1) The search for updating the graph mining results in this invention is an iterative process. In each iteration, a neighboring vertex that meets the pattern constraints is first selected, considering the current partially matched subgraph U. v The pattern constraints include label constraints and topological constraints. Vertices and edges must have labels that conform to the pattern definition, and the connection relationships between vertices in the data graph through edges must be consistent with the query pattern. Initially, U only includes the updated edges. This process requires calling a deduplication function to check whether the match meets the filtering conditions for redundancy exploration and duplicate matching. The specific filtering conditions will be described later. It should be noted that in the actual result set, an affected result may contain added, deleted, or both edges. The checking logic in the deduplication function must also ensure that an affected result can only be generated by one updated edge.

[0054] (3.2) Next, the algorithm adds vertices to U. v as well as v The edges between the graph and existing vertices in U expand the subgraph U into U′. Next, this invention needs to determine whether the expanded subgraph U′ is a new matching result generated by the updated edges, and whether future expansions of U′ might still contain matches. If the latter condition is true, the exploration function is recursively called. Otherwise, backtracking is performed and another adjacent vertex is tried. The exploration process ends after all adjacent vertices of the updated edges have been explored. This ensures that all subgraphs containing the graph update and matching the pattern constraints have been explored. Under this basic processing flow, the computational kernel of this invention aims to ensure that after each update, no matching result is missed, i.e., the update result is exactly the same as the result obtained by running the static algorithm on the graph data before and after the update and then comparing the two results. This needs to be ensured by checking U′ after each expansion.

[0055] The core of checking the subgraph U′ expanded from the updated edges lies in generating the subgraph structure formed by the vertices in U′ before the graph update, and determining whether the subgraphs before and after the update can form a match. This is the goal of the update result matching technique described above. If they can, it is further determined whether the match is newly created or removed. To obtain the previous version of the subgraph U′, it is necessary to compute the subgraph corresponding to the vertices within U′ in the previous graph snapshot. This previous version can be generated by undoing the update edges known to be included (removing added edges or adding deleted edges).

[0056] For each set of pre-update and post-update subgraphs, this invention checks whether the subgraph needs and can be further expanded. If true, it then checks whether the subgraph is connected and meets the query pattern constraints. If the pre-update subgraph passes all tests, it means that it corresponds to a match that existed before the graph update but no longer exists due to this update, and the current subgraph must output it as the removed match. Similarly, if the post-update subgraph passes all tests, the corresponding match existed in the graph after the graph update and should be output as a newly created match (added). Furthermore, if either the pre-update or post-update subgraph passes the filter, the exploration function needs to be recursively called. To meet the correctness criterion, expansion must continue as long as at least one version of the subgraph can be expanded (because it may still produce matches). Otherwise, the exploration will backtrack.

[0057] Regarding load balancing, this invention ensures that the exploration tasks for each update are independent and can be executed in any order. This independence is achieved by combining change detection and deduplication mechanisms. This feature allows multiple work units to perform exploration on non-overlapping updates, and even for updates within the same batch, the behavior remains the same as when this algorithm is executed sequentially on a single work unit. Each work unit executes the exploration algorithm independently. Idle work units select the next update from the work queue for processing and output the corresponding changes in the matching set. Work units obtain the structure and label information required to execute the exploration algorithm by accessing graph storage.

[0058] This invention employs multiple concurrent FIFO queues to implement a work queue, ensuring the persistence of updates and facilitating the submission of results to work units. The work queue provides FIFO semantics, and updates are ordered by timestamp; that is, any operation pulling from the queue guarantees an update with a timestamp no higher than any other update in the queue. A single work queue will not become a performance bottleneck because the processing workload for each update is considerable. In this invention, any work unit can process any update because each work unit can access any part of the input graph from the graph storage. Therefore, there is no need to partition the updates in the queue. As long as there are enough updates, the proposed solution can keep all work units busy while ensuring a balanced load across them.

[0059] In GPU computing, this invention employs multiple task queues instead of a single queue to achieve fine-grained workload management and dynamic adjustment. When a worker thread needs to read a task, it reads from one of the multiple task queues according to a uniform probability distribution. When its task is completed and its assigned task queue is empty, it selects another task queue according to a probability distribution related to the number of tasks in the current queue and takes a task from the head of the queue. The uniform random initial selection ensures that tasks are roughly evenly distributed across multiple queues. When some threads finish early, causing their "local" queues to become empty (i.e., workload imbalance occurs), the probability selection strategy based on the number of tasks immediately guides these idle threads to "support" the queue with the largest task backlog. This is equivalent to implementing a variant of efficient work-stealing, which can continuously and automatically smooth out the differences in busy and idle periods between computing units, ensuring that the GPU's streaming multiprocessors always have tasks available for execution.

[0060] Although the present invention has been described through the above preferred embodiments, those skilled in the art will understand that various modifications in form and detail can be made therein without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention should be defined by the appended claims.

Claims

1. A method for incremental graph mining based on edge update in dynamic graphs of financial transactions, characterized in that, The method includes: Construct a financial transaction graph, where edges in the graph represent the interaction behavior of vertices. The types of vertices include accounts, companies, and consumers, and the interaction behavior includes purchasing goods and transferring funds. Based on several data operations oriented towards the financial transaction graph, a work queue containing several update tasks is generated; wherein, the data operations include edge insertion or deletion operations; Multiple computing cores retrieve update tasks from the work queue in parallel; Graph mining is performed starting from the edges in the data operations, and combined with the snapshot of the financial transaction graph before the update, the difference data corresponding to each update task is obtained. The difference data is aggregated to obtain the graph mining results.

2. The method according to claim 1, characterized in that, The vertex index comprises a hash layer and a vector index layer; the vertex index is used to locate the neighbor list of the vertex in the data layer based on the vertex identifier; the vertex identifier's prefix... The index of the vertex identifier is maintained by a hash layer. Each byte is indexed by a vector index layer. , It is a positive integer, and , The length of the vertex identifier.

3. The method according to claim 2, characterized in that, The vector index layer is an L-layer index tree structure, where each layer indexes one byte; wherein, the vector index layer includes full mapping nodes and / or compressed mapping nodes; the full mapping node contains a length of 2 An array of pointers to a power of 2, wherein the array of pointers to the full-mapped node points to a node in the next level of the index tree or a list of neighbors in the data layer; the compressed mapping node contains a 2 The power of a bit vector and a length of 2. An array of pointers to the power of 1, wherein the array of pointers to the compressed mapping node points to the node of the next level of the index tree or the neighbor list of the data layer, and the bit vector is used to identify the existence status of the node of the next level or the neighbor list of the data layer.

4. The method according to claim 2, characterized in that, If the degree of a vertex is greater than a set threshold, the vertex index of the vertex also includes a fast path for the vertex with a high degree; wherein the fast path for the vertex with a high degree is implemented based on a Bloom filter and maintains a list of pointers to the neighbor list of the vertex.

5. The method according to claim 2, characterized in that, In the case where the data operation is an edge insertion operation, the step of generating a work queue containing several update tasks based on several data operations oriented towards the financial transaction graph includes: Obtain the vertex identifier corresponding to the edge to be inserted, perform multi-level positioning operations based on the vertex identifier, and find the insertion buffer to be written; Using the previous vertex identifier The byte hash value locates the storage address of the first-level node in the vector index layer; based on the last vertex identifier... Calculate the bit vector position corresponding to the vertex identifier byte by byte, layer by layer; If the current node is a compressed mapping node and the corresponding bit vector position is 1, or if the current node is a full mapping node and the corresponding position index pointer is not null, then jump to the next level along the pointer; if the jump condition is not met, then set the corresponding bit vector position of the compressed mapping node to 1, and initialize the next level node and the corresponding pointer. Repeat the hierarchical jump steps and determine the target insertion buffer by using the last node of the vector index layer; Read the tail counter of the target insertion buffer to obtain the write position: if the tail counter has not reached the upper limit of the insertion buffer capacity, write the edge data directly, generate the corresponding update task, and update the element counter and tail counter; if the tail counter has reached the upper limit of the insertion buffer capacity, append the edge data to the sorted data area and generate the corresponding update task.

6. The method according to claim 2, characterized in that, In the case where the data operation is an edge deletion operation, the step of generating a work queue containing several update tasks based on several data operations oriented towards the financial transaction graph includes: Locate the insertion buffer corresponding to the target vertex, and scan the insertion buffer to find the edge to be deleted; If the edge to be deleted is not found in the insertion buffer, the process jumps to the sorted data area and uses a binary search to locate the edge to be deleted. After modifying the data of the edge to be deleted to a preset special deletion marker value, a corresponding update task is generated.

7. The method according to claim 2, characterized in that, The data operations also include edge query operations; wherein the query operations include: Perform multi-level indexing and location operations based on vertex identifiers to find the neighbor list corresponding to the target vertex; If the neighbor list of the target vertex is successfully located, the corresponding edge query operation is performed based on the query requirements: if the query requirement is a neighborhood scan requirement, the insertion buffer and sorted data area are scanned sequentially to obtain the complete neighbor list of the target vertex; if the query requirement is an edge existence check requirement, the insertion buffer is scanned and the sorted data area is searched to locate the target vertex and determine whether the corresponding edge exists.

8. The method according to claim 1, characterized in that, Graph mining starts from the edges in data operations, including: Initialize the matching subgraph based on the edges in the data operations; Select an unmatched vertex from the financial transaction graph that is adjacent to the matching subgraph and satisfies the query pattern constraints; Add the unmatched vertex and the edges between the unmatched vertex and vertices in the matched subgraph to the matched subgraph to obtain the extended subgraph; Within the extended subgraph, undo the edges in the data operation to obtain the subgraph before the update; The algorithm checks the subgraph before and after the update to determine if it can be further expanded. If it can be further expanded, it verifies whether the subgraph's connectivity, vertex and edge labels, and topology satisfy the query pattern constraints. If the subgraph before the update satisfies the constraints, it outputs the removed matching results; if the subgraph after the update satisfies the constraints, it outputs the newly added matching results. When the subgraph before update or the subgraph after expansion has expansion potential, the step of selecting an unmatched vertex from the financial transaction graph that is adjacent to the matching subgraph and satisfies the query pattern constraint is re-executed based on the expanded subgraph. When neither the subgraph before the update nor the subgraph after the expansion has expansion potential, select another unmatched vertex from the financial transaction graph that is adjacent to the matching subgraph and satisfies the query pattern constraint, and re-execute the step of adding the unmatched vertex and the edge between the unmatched vertex and the vertex in the matching subgraph to the matching subgraph to obtain the expanded subgraph.

9. A system for incremental graph mining based on edge update in a dynamic graph of financial transactions, characterized in that, The system includes: The graph construction module is used to construct a financial transaction graph, in which the edges represent the interaction behavior of vertices. The types of vertices include accounts, companies, and consumers, and the interaction behavior includes purchasing goods and transferring funds. The work queue generation module is used to generate a work queue containing several update tasks based on several data operations oriented towards the financial transaction graph; wherein, the data operations include edge insertion or deletion operations; The graph mining module is used by multiple computing kernels to retrieve update tasks from the work queue in parallel; it performs graph mining starting from the edges in the data operations, and combines the snapshot of the financial transaction graph before the update to obtain the difference data corresponding to each update task; it aggregates the difference data to obtain the graph mining results.

10. A computer device, characterized in that, The computer device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the incremental graph mining method based on edge update driven in the dynamic graph of financial transactions as described in any one of claims 1-8.

Images