Hierarchical traversal method based on abstract graph
By generating summary graphs and converting them to HCSR format, combining hypergraphs and corrected edge representation methods, the storage and calculation bottlenecks of traditional graph traversal methods on large-scale graph data are solved, and efficient and accurate graph traversal is achieved.
Patent Information
- Application Number
- CN202510649680.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-20
- Publication Date
- 2025-08-26
AI Technical Summary
Traditional graph traversal methods have storage and computing bottlenecks on large-scale graph data, and sparse matrix encoding is difficult to retain the semantic structure information of the graph, resulting in poor performance in complex query tasks.
By generating a summary graph and converting it into a hierarchical compressed sparse row (HCSR) format, the HCSR representation of the hypergraph is used for hierarchical graph traversal, and the traversal situation is corrected by the HCSR representation of the corrected edges to optimize the data access mode.
It significantly reduces storage space usage, optimizes data access mode, improves the efficiency and accuracy of graph traversal, avoids blind traversal and computing redundancy, and improves the performance of large-scale graph computing.
Smart Images
Figure CN120541273A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of graph processing technology, and specifically to a hierarchical traversal method based on a summary graph. Background Art
[0002] Graph traversal is a core operation in fields such as social network analysis, traffic route planning, and knowledge graph reasoning, and its performance directly affects the efficiency of large-scale graph computing. However, traditional graph traversal methods (such as Breadth-First Search (BFS) and Dijkstra) have significant storage and computational bottlenecks on large-scale graph data. For example, BFS on a graph with tens of billions of edges may cause the memory requirements to exceed the capacity of a single machine, while the query latency of the Dijkstra algorithm on unstructured large graphs may reach minutes. In addition, although sparse matrix encoding such as Compressed Sparse Row (CSR) can reduce storage overhead, it is difficult to preserve the semantic structure information of the graph, resulting in poor performance in complex query tasks (such as cross-community relationship reasoning). Summary of the Invention
[0003] In view of this, an embodiment of the present application provides a hierarchical traversal method based on a summary graph, which can significantly improve the performance and efficiency of graph traversal.
[0004] In a first aspect, a hierarchical traversal method based on a summary graph is provided, the method comprising: generating a summary graph of an original graph, the summary graph comprising a mapping relationship between nodes and supernodes, hyperedge connections, and a set of modified edges in the original graph; converting the summary graph into a hierarchical compressed sparse row (HCSR) format, the summary graph in the HCSR format comprising an HCSR representation of a hypergraph and an HCSR representation of modified edges; performing a hierarchical graph traversal on the original graph based on the HCSR representation of the hypergraph, wherein the processing of the supernode layer connection relationship in the hypergraph precedes the fine-grained access to the subnode layer in the hypergraph; and modifying the traversal situation based on the HCSR representation of the modified edges.
[0005] In a possible implementation, the modifying the traversal situation based on the HCSR representation of the modified edge includes: modifying the traversal situation based on the HCSR representation of the modified edge during the hierarchical graph traversal.
[0006] In a possible implementation, the modifying the traversal situation based on the modified edge HCSR representation includes: after completing the layered graph traversal, modifying the traversal situation based on the modified edge HCSR representation.
[0007] In a possible implementation, generating a summary graph of the original image includes: generating a summary graph of the original image using K-hop clustering, a Min-Hash similarity calculation method, and a minimum description length (MDL) criterion.
[0008] In one possible implementation, the summary graph of the original graph is generated by using K-hop clustering, Min-Hash similarity calculation method and minimum description length MDL criterion, including: performing K-hop clustering on the nodes in the original graph to form a candidate merge set including the nodes and their neighbors; performing Min-Hash signature calculation on any two nodes in the candidate merge set to obtain similarity, and forming a maximum heap set from node pairs whose similarity exceeds a first threshold; calculating the storage cost change of the node pairs in the maximum heap set based on the MDL criterion, and iteratively merging the node pairs whose storage cost change is greater than a second threshold; gradually merging nodes to finally generate a summary graph including the mapping relationship between the nodes and super nodes in the original graph, the hyperedge connection and the modified edge set.
[0009] In one possible implementation, the HCSR representation of the hypergraph includes a supernode offset array, a supernode value array, a child node offset array, and a child node value array, wherein the supernode offset array stores the starting index of each supernode's neighbor supernode, the supernode value array stores the supernode's neighbor supernodes, the child node offset array stores the starting index of each child node in the supernode, and the child node value array stores all child nodes included in the supernode; the HCSR representation of the correction edge includes a positive correction offset array, a positive correction value array, a negative correction offset array, and a negative correction value array, wherein the positive correction offset array stores the starting index of the positive correction edge of each child node, the positive correction value array stores the target child nodes of all positive correction edges, the negative correction offset array stores the starting index of the negative correction edge of each child node, and the negative correction value array stores the target child nodes of all negative correction edges.
[0010] In one possible implementation, the layered graph traversal of the original graph based on the HCSR representation of the hypergraph and the correction of the traversal based on the HCSR representation of the correction edge include: loading a supernode layer into a graphics processing unit (GPU), initializing a frontier queue, adding an initial node to the frontier queue, and using it as the starting point of the first iteration; checking whether any positive correction edge in the HCSR representation of the correction edge is associated with the starting point; if so, marking the node associated with the starting point in the positive correction edge as a neighbor node of the starting point, and adding the neighbor node of the starting point to an output queue for use in the next iteration; determining the supernode to which the starting point belongs based on the HCSR representation of the hypergraph, searching for and traversing all neighboring supernodes of the supernode to which the starting point belongs, marking all child nodes included therein, and adding the marked nodes to the output queue for use in the next iteration; and checking whether any negative correction edge in the HCSR representation of the correction edge is associated with the starting point; if so, deleting the node associated with the starting point in the negative correction edge from the output queue.
[0011] In a second aspect, a hierarchical traversal device based on a summary graph is provided, comprising: a summary graph generation module for generating a summary graph of an original graph, the summary graph comprising a mapping relationship between nodes and supernodes, hyperedge connections, and a set of modified edges in the original graph; an HCSR data format conversion module for converting the summary graph into a hierarchical compressed sparse row (HCSR) format, the summary graph in the HCSR format comprising an HCSR representation of a hypergraph and an HCSR representation of modified edges; a graph traversal module for performing a hierarchical graph traversal on the original graph based on the HCSR representation of the hypergraph, wherein the processing of the supernode layer connection relationship in the hypergraph precedes the fine-grained access of the subnode layer in the hypergraph; and a correction module for correcting the traversal situation based on the HCSR representation of the modified edges.
[0012] In a third aspect, a computer device is provided, comprising: a processor, and a memory communicatively connected to the processor, wherein the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the hierarchical traversal method based on the summary graph of the first aspect and any possible implementation of the first aspect.
[0013] In a fourth aspect, a computer-readable storage medium is provided, which includes a stored program, wherein when the program is run, the device where the computer-readable storage medium is located is controlled to execute a hierarchical traversal method based on a summary graph such as the first aspect and any possible implementation method of the first aspect.
[0014] Based on the above technical solutions, first, by generating a summary graph containing node and supernode mapping, hyperedge connections and modified edge sets, the complex structure of the original graph is abstracted and refined, reducing the complexity of data processing; the summary graph is converted into the Hierarchical Compressed Sparse Row (HCSR) format, which not only greatly reduces the storage space occupied, but also optimizes the data access mode through the separate representation of the hypergraph and the modified edges. The layered graph traversal strategy prioritizes the connection relationship at the supernode layer, and then performs fine-grained access at the subnode layer, realizing an efficient exploration path from coarse to fine, avoiding blind traversal, and significantly reducing computational redundancy. The traversal situation is corrected based on the HCSR representation of the modified edges, ensuring the accuracy of the node relationship during the layered traversal process, effectively compensating for the information loss caused by the hypergraph abstraction, and improving the integrity and reliability of the traversal results. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the drawings without creative work.
[0016] Figure 1 A schematic block diagram of a hierarchical traversal method based on a summary graph according to an embodiment of the present application is shown.
[0017] Figure 2 Another schematic block diagram of the summary graph-based hierarchical traversal method according to an embodiment of the present application is shown.
[0018] Figure 3 A schematic diagram showing the original image and summary image of an embodiment of the present application.
[0019] Figure 4 A schematic diagram showing a summary diagram of the HCSR format according to an embodiment of the present application.
[0020] Figure 5 Another schematic block diagram of the hierarchical traversal method based on the summary graph according to an embodiment of the present application is shown.
[0021] Figure 6 A schematic diagram of the graph traversal and correction process of an embodiment of the present application is shown.
[0022] Figure 7 A schematic block diagram of a hierarchical traversal device based on a summary graph according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0023] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0024] Graph traversal is a core operation in fields such as social network analysis, traffic route planning, and knowledge graph reasoning, and its performance directly affects the efficiency of large-scale graph computing. However, traditional graph traversal methods (such as Breadth-First Search (BFS) and Dijkstra) have significant storage and computational bottlenecks on large-scale graph data. For example, BFS on a graph with tens of billions of edges may cause the memory requirements to exceed the capacity of a single machine, while the query latency of the Dijkstra algorithm on unstructured large graphs may reach minutes. In addition, although sparse matrix encoding such as Compressed Sparse Row (CSR) can reduce storage overhead, it is difficult to preserve the semantic structure information of the graph, resulting in poor performance in complex query tasks (such as cross-community relationship reasoning).
[0025] Graph summarization is an innovative method for compressing large-scale graph data. It combines structurally similar nodes and edges to construct a more compact summary graph, reducing storage requirements and accelerating computation. Existing summary graph methods (such as SWeG and SAGS) are primarily used for pattern mining and static analysis, but are not deeply integrated with efficient traversal algorithms, making them difficult to leverage in practical graph query tasks. Furthermore, the summary graph construction process requires a balance between compression ratio and information preservation, and existing methods often fail to simultaneously meet the requirements for high compression ratio and traversal efficiency.
[0026] With the widespread use of Graphics Processing Units (GPUs) and multi-core processors, graph traversal optimization based on parallel computing has become a research hotspot. Existing graph processing frameworks use GPUs to accelerate graph traversal, but there is still room for improvement in load balancing and storage optimization. Parallel traversal algorithms usually rely on efficient data structures, such as the Compressed Sparse Row (CSR) format. However, when the graph has a hierarchical structure (such as supernode-child node relationships), the traditional CSR format cannot effectively express the multi-level characteristics of the graph, resulting in reduced traversal efficiency. Therefore, how to combine efficient data structures and parallel computing to realize a graph processing framework that can both maintain semantic information and achieve efficient traversal has become a major challenge in large-scale graph traversal optimization.
[0027] In view of this, the embodiment of the present application provides a hierarchical traversal method based on a summary graph. Through the summary graph and the HCSR data format, it is possible to effectively retain the key structural information of the original graph (such as community structure, frequent paths, etc.) when compressing the graph, avoiding the problem of losing the graph structure semantics in traditional graph compression methods, thereby making full use of this structural information during traversal and improving the efficiency and accuracy of traversal. In addition, the hierarchical graph traversal algorithm based on the HCSR format starts from the supernode layer and goes down layer by layer to the subnode layer, using the hierarchical information of the graph to quickly locate the effective access path, avoiding unnecessary traversal operations, and significantly improving the computational efficiency of graph traversal, especially when processing large-scale graphs.
[0028] Figure 1 Schematic block diagram of the hierarchical traversal method based on the summary graph of an embodiment of the present application is shown. Optionally, the method can be executed by a processor, for example, by a GPU. Figure 1 As shown, the method 100 includes part or all of the following contents.
[0029] S110 , generating a summary graph of the original graph, wherein the summary graph includes mapping relationships between nodes and supernodes in the original graph, superedge connections, and a modified edge set.
[0030] S120 , converting the summary graph into a hierarchical compressed sparse row (HCSR) format, wherein the summary graph in the HCSR format includes an HCSR representation of a hypergraph and an HCSR representation of a modified edge.
[0031] S130 , performing a hierarchical graph traversal on the original graph based on the HCSR representation of the hypergraph, wherein processing of a super-node layer connection relationship in the hypergraph precedes fine-grained access to a sub-node layer in the hypergraph.
[0032] S140: Modify the traversal situation based on the HCSR representation of the modified edge.
[0033] In S110, first, the nodes in the original graph can be divided into multiple super nodes according to certain rules. The division rules can be based on the attributes, connection relationships or other relevant information of the nodes. For example, the nodes can be divided according to their degrees (i.e., the number of edges connected to the node), and nodes with similar degrees can be divided into the same super node; or graph clustering algorithms such as spectral clustering, hierarchical clustering, etc. can be used to divide the graph into multiple closely connected sub-graphs, each sub-graph as a super node. After the super node division is completed, it is necessary to establish a mapping relationship between each node in the original graph and the super node. An array can be used to store this mapping relationship, where the index represents the node in the original graph and the value represents the super node to which the node belongs.
[0034] Secondly, a hyperedge is an edge connecting multiple supernodes, indicating a relationship between these supernodes. When constructing a hyperedge, the connectivity between nodes in the original graph must be considered. Specifically, all edges in the original graph are traversed, and for each edge, the supernodes to which its two endpoints belong are found. If these two supernodes are different, a hyperedge connecting them is added to the hypergraph. A list or adjacency list can be used to store hyperedge information.
[0035] Finally, since the hypergraph is an abstraction of the original graph, some important information may be lost or some information that is not in the original graph may be added. The correction edge set is used to record this lost or added information so that it can be corrected in subsequent processing. For example, traverse all the edges in the original graph. For each edge, if its two endpoints belong to the same supernode, but there is no corresponding edge inside the supernode in the hypergraph to represent this connection relationship, then this edge is added to the correction edge set as a correction edge. Similarly, a list can be used to store correction edge information. Optionally, the correction edge set includes positive correction edges and negative correction edges. Positive correction edges refer to edges that exist in the original graph but are not directly reflected in the hypergraph due to the hypergraph abstraction process (such as node aggregation to form a supernode) and need to be added to the summary graph correction edge set. Negative correction edges refer to "false" edges generated by the abstraction strategy during the hypergraph construction process, that is, connection relationships that do not exist in the original graph. They need to be clearly recorded in the summary graph correction edge set and excluded in subsequent processing.
[0036] The mapping relationship between nodes and super nodes in the original graph, the hyperedge connections, and the modified edge set are combined to form the final summary graph. A data structure, such as a dictionary or class, can be used to store all the information of the summary graph.
[0037] In some embodiments, S110, ie, generating a summary graph of the original image, includes: generating a summary graph of the original image using K-hop clustering, a Min-Hash similarity calculation method, and a minimum description length (MDL) criterion.
[0038] Specifically, if Figure 2 As shown, the S110 includes generating a summary image of the original image using K-hop clustering, Min-Hash similarity calculation method and minimum description length MDL criterion, and further includes part or all of the following contents.
[0039] S111 , performing K-hop clustering on the nodes in the original graph to form a candidate merge set including the nodes and their neighbors.
[0040] S112: Perform Min-Hash signature calculation on any two nodes in the candidate merge set to obtain similarity, and form a maximum heap set from node pairs whose similarity exceeds a first threshold.
[0041] S113 , calculating a storage cost change of merging node pairs in the maximum heap set based on the MDL criterion, and iteratively merging node pairs whose storage cost change is greater than a second threshold.
[0042] S114 , gradually merging nodes to eventually generate a summary graph including the mapping relationship between the nodes and supernodes in the original graph, the superedge connections, and the modified edge set.
[0043] In S111, K-hop clustering is a clustering method based on node neighborhood information. For each node in the original graph, all neighboring nodes within a K-hop radius are found, with that node as the center. "K-hops" here refers to all nodes that can be reached from the central node via a maximum of K edges. Each node and its neighboring nodes within a K-hop radius are combined to form a set. All such sets constitute a candidate merge set. This set contains node pairs that are potentially suitable for merging because they have relatively close connections in the graph.
[0044] In S112, Min-Hash is a technology for quickly estimating the similarity between two sets. For any two nodes in the candidate merge set, their respective neighbor node sets are regarded as two sets. Signatures are generated for the two sets using the Min-Hash algorithm, and the signature is a sequence of hash values. The two signatures are then compared to calculate their similarity. The calculation of similarity is usually based on Jaccard similarity, that is, the size of the intersection of two sets divided by the size of the union. A first threshold is set to filter out node pairs whose similarity exceeds the threshold. These node pairs indicate that their neighbor structures in the graph have a high similarity and are likely to be merged. These node pairs are constructed into a maximum heap in descending order of similarity to form a maximum heap set. The characteristics of the maximum heap make it possible to quickly obtain the node pairs with the highest similarity each time.
[0045] In S113, the Minimum Description Length (MDL) criterion is a criterion for evaluating model complexity and data fitting degree. In graph data processing, it can be used to measure the change in storage cost of the graph after the node pairs are merged. The storage cost is usually related to the representation of the graph (such as adjacency matrix, adjacency list) and the number of nodes and edges. For each node pair in the maximum heap set, the difference between their storage cost after merging and the storage cost before merging is calculated to obtain the storage cost change. This change reflects the impact of the merge operation on the storage cost of the graph. A second threshold is set, and node pairs with a storage cost change greater than the threshold are merged. The merge operation updates the structure of the graph, including node and edge information. After each merge, the similarity and storage cost change of the relevant node pairs are recalculated, and the maximum heap set is updated. Repeat this process until there are no node pairs that meet the conditions and can be merged.
[0046] In S114, during the iterative merging process of S113, the node pairs that meet the conditions are continuously merged into a super node. As the merging proceeds, the number of nodes in the graph gradually decreases, forming some super nodes composed of multiple original nodes. During the node merging process, the edge information needs to be updated. For edges connecting different super nodes, they are retained and used as hyper edges. A list or adjacency list can be used to store hyper edge information. Since node merging may cause some edge information to be lost or require correction, a set of corrected edges needs to be determined. The corrected edge set is used to record those edges that require additional processing in the hyper graph to ensure the information integrity of the graph. The mapping relationship between nodes and super nodes, hyper edge connections, and the corrected edge set are combined to form the final summary graph. A data structure, such as a dictionary or a class, can be used to store all the information of the summary graph.
[0047] In one embodiment, the summary image is a compressed representation of the original image. Figure 3 As shown in (a), after being processed by the graph summary algorithm, the corresponding summary graph is generated, as shown in Figure 3 As shown in (b). Nodes 1-5 in the original graph are highly similar in structure, so supernode S1 clusters these five structurally similar subnodes together, thereby reducing storage overhead. The hyperedge between S1 and S2 replaces the five subedges between nodes 1-5 and node 6 in the original graph, thereby improving the connectivity and expressiveness of the graph. The introduction of corrected edges effectively handles complex adjacency relationships, where C+ represents the forward corrected edge set, which represents edges that exist in the original graph but not in the summary graph. For example, edge (6, 7) exists in the original graph, but in order to reduce storage overhead, the summary graph does not save this edge in the hypergraph but saves it in the corrected edge C+. C- represents the reverse corrected edge set, which represents edges that do not exist in the original graph but in the summary graph. The existence of C- is similar to that of C+, so it will not be repeated.
[0048] Specifically, from the perspective of storage overhead, the original graph consists of 10 edges, while the summary graph consists of only 6 edges. Therefore, this summary graph representation can not only effectively compress the edge set of the original graph, but also retain key topological features, thereby providing higher computational efficiency in the analysis and processing of the graph. Especially in the calculation of large-scale graphs, it can greatly reduce the computational burden and memory consumption during the traversal process.
[0049] In one of the embodiments, as the scale of the graph continues to grow, the graph summary algorithm has difficulty processing large-scale graphs, and its application field is relatively limited, but its wide application in pattern mining reveals its huge potential in the field of graph traversal. The embodiment of the present application generates a summary graph through K-hop clustering, Min-Hash similarity calculation and MDL criterion to reduce redundant data and reduce storage overhead, while retaining the core structural information of the graph for application in the field of graph traversal.
[0050] In an embodiment where K = 2, the 1-hop and 2-hop neighbors of each node in the graph are first determined to form candidate node pairs. These node pairs are then used to perform Min-Hash similarity calculations. K-hop clustering captures the relationships between each node and its neighbors and the local structure of the graph, generating a potential candidate set for subsequent similarity calculations and node merging. This step lays the foundation for generating a summary graph, ensuring a high degree of local consistency during the merging process.
[0051] In one embodiment, within the candidate set, a Min-Hash signature calculation is performed on any pair of candidate nodes. By calculating the similarity of each pair of nodes, it is possible to determine which node pairs are structurally similar. A threshold is set, for example, 0.7, and all node pairs with a similarity exceeding this threshold are inserted into a max heap set to facilitate subsequent rapid access to these highly similar node pairs. This step effectively reduces the complexity of the similarity calculation by using the Min-Hash signature calculation, while simultaneously screening out node pairs that contribute to the summary graph.
[0052] In one embodiment, the change in storage cost after merging each pair of candidate nodes is calculated based on the MDL criterion. By calculating the storage cost of the node pairs before and after the merger, the node pairs with a storage cost change greater than 0 are selected for merging. Optionally, the merging cost of each pair of nodes A and B is calculated, i.e., Reduction(A,B)=Cost(A)+Cost(B)-Cost(A∪B), and the node pairs with increased merging costs are screened out for merging iterations. In this way, the MDL criterion helps ensure that the storage gain of the merged node pairs is minimized, thereby maintaining the compression effect and information quality of the graph.
[0053] In one embodiment, nodes are gradually merged until a final summary graph is formed. By continuously performing node merging operations, node pairs with increasing merging costs are selected for merging each time, and finally a summary graph containing super nodes, hyper edges, and corrected edges (C+, C-) is generated. Specifically, after each merge, the new node (super node) formed represents the original node set after the merge, and the hyper edge represents the connection relationship between these nodes. The corrected edge is used to represent the changes in the adjacency relationship that may occur after the merge operation. The final generated summary graph is more compact than the original graph, can effectively reduce storage overhead, and retain the core structural information of the graph, which is suitable for subsequent graph traversal or analysis tasks.
[0054] In S120, first, the representation of supernodes and superedges in the summary graph is converted to a double-layer CSR (HCSR) format to generate a four-tuple structure. The supernode offset array stores the starting index of each supernode's neighboring supernodes; the supernode value array stores the supernode's neighboring supernodes; the child node offset array stores the starting index of each child node within the supernode; and the child node value array stores all child nodes included in the supernode. Secondly, the modified edge representation in the summary graph is converted to a double-layer CSR (HCSR) format for efficient storage and access of modified edge data, generating a four-tuple structure. The positive modified (C+) offset array stores the starting index of the C+ modified edge of each child node; the C+ value array stores the target child nodes of all C+ modified edges; the negative modified (C-) offset array stores the starting index of the C- modified edge of each child node; and the C- value array stores the target child nodes of all C- modified edges.
[0055] In one embodiment, Figure 4 As shown in (a), the summary graph consists of three supernodes (S1, S2, and S3), whose global indexes are 0, 1, and 2, respectively. Taking S2, whose supernode index is 1, as an example, in the supernode offset array, S2's starting index is 1 and its ending index is 3 (defined using a left-closed, right-open interval), indicating that S2's adjacent supernode data is stored in indexes 1 to 2 of the supernode value array. Parsing this array content reveals that S2's adjacent supernodes are S1 (index 1) and S3 (index 2). Furthermore, in the child node offset array, supernode S1's starting index is 1 and its ending index is 6, indicating that it contains five child nodes (index difference: 6-1=5). By querying indexes 1 to 5 of the child node value array, we obtain its internal child node ID set {1, 2, 3, 4, 5}, thus fully mapping S1's supernode-child node affiliation.
[0056] In one embodiment, Figure 4As shown in (b), the original graph consists of 10 child nodes, whose global indexes are 1-10. Taking the node with node index 6 as an example, in the C+ offset array, the starting index of the node is 0 and the ending index is 1 (defined using the left-closed and right-open interval), indicating that the adjacent node data of the node in C+ is stored in the 0th to 1st index positions of the C+ value array. According to the analysis of the array content, the adjacent node of the node in C+ is node 7. In addition, in the C- offset array, the starting index of the node is 0 and the ending index is 2, indicating that it has 2 adjacent child nodes in C- (index position difference: 2-0=2). By querying the index positions 0 to 2 of the C- value array, its adjacent child node set {9, 10} in C- is obtained, and the edge-child node affiliation relationship is completely mapped and corrected.
[0057] In some embodiments, S140, namely, correcting the traversal situation based on the HCSR representation of the corrected edge, includes: during the hierarchical graph traversal, correcting the traversal situation based on the HCSR representation of the corrected edge.
[0058] Alternatively, as Figure 5 As shown, S140 includes, during the layered graph traversal process, modifying the traversal situation based on the HCSR representation of the modified edge, and further includes part or all of the following contents.
[0059] S141, loading the super node layer into the graphics processing unit (GPU), initializing the frontier queue, adding the initial node into the frontier queue, and using it as the starting point of the first iteration;
[0060] S142, checking whether any forward modified edge in the HCSR representation of the modified edge is associated with the starting point; if so, marking the node associated with the starting point in the forward modified edge as a neighbor node of the starting point, and adding the neighbor node of the starting point to an output queue for use in the next iteration;
[0061] S143, based on the HCSR representation of the hypergraph, determining the supernode to which the starting point belongs, searching and traversing all neighboring supernodes of the supernode to which the starting point belongs, marking all child nodes contained therein, and adding the marked nodes to the output queue for use in the next iteration;
[0062] S144, checking whether there is a negative modified edge in the HCSR representation of the modified edge associated with the starting point; if so, deleting the node in the negative modified edge associated with the starting point from the output queue.
[0063] Specifically, during the initialization phase, the supernode hierarchy is loaded onto the GPU, the frontier queue is initialized, and the initial node is added to the queue as the frontier node set for the first iteration. Next, the forward corrected edges of the currently traversed child node are processed, and the neighbors of the currently traversed child node are quickly located using the HCSR structure. Each thread block loads the C+ offset array associated with the node it is processing, reducing global memory access latency. During this process, the thread marks the neighboring nodes as visited and adds them to the output queue. Next, using the supernode information in the HCSR structure, the list of neighboring supernodes and their child nodes of the supernode containing the currently traversed child node is quickly retrieved. This step utilizes warp-level parallelism, with each warp responsible for processing a supernode and assigning its multiple child nodes to the 32 threads within the warp. Each thread traverses its responsible child nodes and adds the visited child nodes to the output queue. By caching supernode and adjacency information in shared memory, global memory access latency is further reduced, improving overall traversal efficiency. Finally, the negative corrected edges of the currently traversed child node are processed. Through the HCSR data structure, the neighbors of the currently traversed child node in the negative corrected edge array are quickly obtained. C- is divided into multiple segments according to the node ID hash value. Each segment is processed by an independent thread block, so that each thread only processes the data within a specific segment and removes the corresponding node from the output queue.
[0064] In one embodiment, Figure 6 As shown in (a), the HCSR structure is loaded into the GPU, the frontier queue is initialized, and the initial node 6 is added to the frontier queue as the starting point of the first iteration.
[0065] In one embodiment, Figure 6 As shown in (b), check whether there is an edge associated with the source node (node 6) in the forward correction edge C+, mark node 7 as a direct neighbor of the source node through the edge (6,7) in C+, and add node 7 to the output queue for use in the next iteration.
[0066] In one embodiment, Figure 6 As shown in Figures (c)-(e), the algorithm determines the supernode to which the source node belongs. For example, node 6 belongs to supernode {6}. Next, it searches for all neighboring supernodes of that supernode. For example, neighboring supernodes include S1 {1, 2, 3, 4, 5} and S2 {8, 9, 10}. It then traverses these neighboring supernodes and marks all of their child nodes. For example, when traversing to supernode S1, it accesses and marks its child nodes {1, 2, 3, 4, 5} based on the child node index stored in the HCSR data structure. The same process is repeated when traversing supernode S2. The marked nodes are added to the output queue for use in the next iteration.
[0067] In one embodiment, Figure 6 As shown in (f), the algorithm checks for negative corrected edges C-, for example, through edges (6,9) and (6,10), removing nodes 9 and 10 from the set of visited nodes. Through these steps, the algorithm completes one iteration and updates the frontier queue, looping through the traversal until all nodes are visited.
[0068] In some other embodiments, S140, namely, correcting the traversal situation based on the HCSR representation of the corrected edges, includes: after completing the layered graph traversal, correcting the traversal situation based on the HCSR representation of the corrected edges.
[0069] In some embodiments, the method 100 further includes: achieving load balancing among GPU threads through a dynamic task stealing strategy, wherein the strategy includes a two-tier task allocation mechanism of a local queue and a global queue.
[0070] Specifically, a local queue is established to store the currently processed subnode interval, with an interval granularity of 64 nodes / unit. The local queue divides tasks using 64 nodes as the basic processing unit. Each GPU thread block is bound to a local queue to store the currently processed subnode interval. This design achieves computation-memory overlap through a double-buffered prefetch mechanism: while a thread processes a subnode of the current interval, the Streaming Multiprocessor (SM) background asynchronously preloads node data for the next interval into shared memory. The global queue then uses a sharded hash table structure to store the ID of the supernode to be processed and its subnode interval index. The hash table is divided into 128 shards, each associated with an independent memory pool and mutex lock. When inserting a task, the target shard is selected based on the supernode ID hash value to avoid global lock contention. When extracting a task, the shard with the highest spatial locality to the current thread block is prioritized (through a NUMA-aware policy). Finally, an idle thread block retrieves a task from the global queue using the atomic CAS operation, with stealing priority sorted by spatial locality. The idle thread block steals tasks from the global queue through the atomic compare-and-exchange (atomicCAS) operation. The specific process is to first read the head pointer of the target shard and verify the task status; if the task is not occupied, the task is marked as "in process" through atomicCAS; after successful theft, the tasks are sorted by spatial locality priority (tasks in the same GPU chip and the same storage node are given priority).
[0071] By introducing a task-stealing mechanism, we reduce inter-thread disparity and ensure load balancing, thereby improving the performance of parallel computing. Combined with the rule-based access method of the HCSR format, it can efficiently process large-scale graph data and significantly improve the performance of graph traversal on parallel architectures, especially when processing very large graphs.
[0072] The embodiment of the present application also provides a hierarchical traversal device based on a summary graph. Figure 7 As shown, the hierarchical traversal device 200 includes part or all of the following contents.
[0073] The summary graph generation module 210 is configured to generate a summary graph of the original graph, wherein the summary graph includes mapping relationships between nodes and supernodes, superedge connections, and a modified edge set in the original graph.
[0074] The HCSR data format conversion module 220 is configured to convert the summary graph into a hierarchical compressed sparse row (HCSR) format. The summary graph in the HCSR format includes an HCSR representation of a hypergraph and an HCSR representation of a modified edge.
[0075] The graph traversal module 230 is configured to perform a hierarchical graph traversal on the original graph based on the HCSR representation of the hypergraph, wherein the processing of the super-node layer connection relationship in the hypergraph precedes the fine-grained access of the sub-node layer in the hypergraph.
[0076] The correction module 240 is configured to correct the traversal situation based on the HCSR representation of the corrected edge.
[0077] In some embodiments, the graph traversal module 230 is specifically configured to: during the hierarchical graph traversal, modify the traversal situation based on the HCSR representation of the modified edges.
[0078] In some embodiments, the graph traversal module 230 is specifically configured to: after completing the hierarchical graph traversal, modify the traversal condition based on the HCSR representation of the modified edge.
[0079] In some embodiments, the summary graph generation module 210 is specifically configured to generate a summary graph of the original image using K-hop clustering, a Min-Hash similarity calculation method, and a minimum description length (MDL) criterion.
[0080] In some embodiments, the summary graph generation module 210 is specifically configured to:
[0081] Performing K-hop clustering on the nodes in the original graph to form a candidate merge set including the nodes and their neighbors;
[0082] Performing Min-Hash signature calculation on any two nodes in the candidate merge set to obtain similarity, and forming a maximum heap set from node pairs whose similarity exceeds a first threshold;
[0083] Calculating a storage cost change of merging node pairs in the maximum heap set based on the MDL criterion, and iteratively merging node pairs whose storage cost change is greater than a second threshold;
[0084] Nodes are merged step by step to finally generate a summary graph including the mapping relationship between nodes and supernodes in the original graph, the superedge connections, and the modified edge set.
[0085] In some embodiments, the HCSR representation of the hypergraph includes a supernode offset array, a supernode value array, a child node offset array, and a child node value array, wherein the supernode offset array stores the starting index of each supernode's neighbor supernode, the supernode value array stores the supernode's neighbor supernodes, the child node offset array stores the starting index of each child node in the supernode, and the child node value array stores all child nodes included in the supernode; the HCSR representation of the correction edge includes a positive correction offset array, a positive correction value array, a negative correction offset array, and a negative correction value array, wherein the positive correction offset array stores the starting index of the positive correction edge of each child node, the positive correction value array stores the target child nodes of all positive correction edges, the negative correction offset array stores the starting index of the negative correction edge of each child node, and the negative correction value array stores the target child nodes of all negative correction edges.
[0086] In some embodiments, the graph traversal module 230 and the correction module 240 are specifically configured to:
[0087] Loading the super node layer to the graphics processing unit (GPU), initializing the frontier queue, adding the initial node to the frontier queue, and using it as the starting point of the first iteration;
[0088] Check whether any forward modified edge in the HCSR representation of the modified edge is associated with the starting point; if so, mark the node associated with the starting point in the forward modified edge as a neighbor node of the starting point, and add the neighbor node of the starting point to an output queue for use in the next iteration;
[0089] Based on the HCSR representation of the hypergraph, determine the supernode to which the starting point belongs, search and traverse all neighboring supernodes of the supernode to which the starting point belongs, mark all child nodes included therein, and add the marked nodes to the output queue for use in the next iteration;
[0090] Check whether there is a negative revision edge in the HCSR representation of the revision edge associated with the starting point; if so, delete the node associated with the starting point in the negative revision edge from the output queue.
[0091] Based on the same concept, an embodiment of the present application further provides a computer device, which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the program instructions are executed by the processor, the computer device executes the various method embodiments described above.
[0092] It should be noted that the details of the device side embodiment can be found in the method side embodiment, and for the sake of brevity, they will not be repeated here.
[0093] Based on the same idea, an embodiment of the present application also provides a computer-readable storage medium, which includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned various method embodiments.
[0094] Although the present application has been described with reference to preferred embodiments, various modifications may be made thereto and components may be substituted with equivalents without departing from the scope of the present application. In particular, the various technical features described in the various embodiments may be combined in any manner as long as there are no structural conflicts. The present application is not limited to the specific embodiments disclosed herein, but encompasses all technical solutions within the scope of the claims.
Claims
1. A hierarchical traversal method based on a summary graph, characterized in that: include: Generate a summary graph of the original graph, the summary graph including mapping relationships between nodes and supernodes in the original graph, superedge connections, and a modified edge set; Converting the summary graph into a hierarchical compressed sparse row (HCSR) format, wherein the summary graph in the HCSR format includes an HCSR representation of a hypergraph and an HCSR representation of a modified edge; Performing a hierarchical graph traversal on the original graph based on the HCSR representation of the hypergraph, wherein processing of the super-node layer connection relationship in the hypergraph precedes fine-grained access to the sub-node layer in the hypergraph; The traversal situation is modified based on the HCSR representation of the modified edge.
2. The method according to claim 1, characterized in that The modifying of the traversal situation based on the HCSR representation of the modified edge includes: During the hierarchical graph traversal, the traversal situation is modified based on the HCSR representation of the modified edge.
3. The method according to claim 1, characterized in that The traversal situation is modified based on the HCSR representation of the modified edge, including: After the layered graph traversal is completed, the traversal situation is modified based on the HCSR representation of the modified edge.
4. The method according to claim 1, wherein Generating a summary image of the original image includes: The summary image of the original image is generated by using K-hop clustering, Min-Hash similarity calculation method and minimum description length MDL criterion.
5. The method according to claim 4, characterized in that The method of generating a summary image of the original image by using K-hop clustering, Min-Hash similarity calculation method and minimum description length (MDL) criterion includes: Performing K-hop clustering on the nodes in the original graph to form a candidate merge set including the nodes and their neighbors; Performing Min-Hash signature calculation on any two nodes in the candidate merge set to obtain similarity, and forming a maximum heap set from node pairs whose similarity exceeds a first threshold; Calculating a storage cost change of merging node pairs in the maximum heap set based on the MDL criterion, and iteratively merging node pairs whose storage cost change is greater than a second threshold; Nodes are merged step by step to finally generate a summary graph including the mapping relationship between nodes and supernodes in the original graph, the superedge connections, and the modified edge set.
6. The method according to claim 1, characterized in that The HCSR representation of the hypergraph includes a supernode offset array, a supernode value array, a child node offset array, and a child node value array, wherein the supernode offset array stores the starting index of the neighboring supernodes of each supernode, the supernode value array stores the neighboring supernodes of the supernode, the child node offset array stores the starting index of the child nodes in each supernode, and the child node value array stores all the child nodes included in the supernode; The HCSR representation of the correction edge includes a positive correction offset array, a positive correction value array, a negative correction offset array, and a negative correction value array, wherein the positive correction offset array stores the positive correction edge starting index of each child node, the positive correction value array stores the target child nodes of all positive correction edges, the negative correction offset array stores the negative correction edge starting index of each child node, and the negative correction value array stores the target child nodes of all negative correction edges.
7. The method according to claim 1, characterized in that The layered graph traversal of the original graph based on the HCSR representation of the hypergraph, and the correction of the traversal based on the HCSR representation of the corrected edge, include: Loading the super node layer to the graphics processing unit (GPU), initializing the frontier queue, adding the initial node to the frontier queue, and using it as the starting point of the first iteration; Check whether any forward modified edge in the HCSR representation of the modified edge is associated with the starting point; if so, mark the node associated with the starting point in the forward modified edge as a neighbor node of the starting point, and add the neighbor node of the starting point to an output queue for use in the next iteration; Based on the HCSR representation of the hypergraph, determine the supernode to which the starting point belongs, search and traverse all neighboring supernodes of the supernode to which the starting point belongs, mark all child nodes included therein, and add the marked nodes to the output queue for use in the next iteration; Check whether there is a negative revision edge in the HCSR representation of the revision edge associated with the starting point; if so, delete the node associated with the starting point in the negative revision edge from the output queue.
8. A hierarchical traversal device based on a summary graph, characterized in that: include: A summary graph generation module is used to generate a summary graph of the original graph, wherein the summary graph includes a mapping relationship between nodes and supernodes in the original graph, superedge connections, and a modified edge set; An HCSR data format conversion module, configured to convert the summary graph into a hierarchical compressed sparse row (HCSR) format, wherein the summary graph in the HCSR format includes an HCSR representation of a hypergraph and an HCSR representation of a modified edge; A graph traversal module, configured to perform a hierarchical graph traversal on the original graph based on the HCSR representation of the hypergraph, wherein the processing of the super-node layer connection relationship in the hypergraph precedes the fine-grained access of the sub-node layer in the hypergraph; The correction module is used to correct the traversal situation based on the HCSR representation of the correction edge.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed, the device where the computer-readable storage medium is located is controlled to implement the method according to any one of claims 1 to 7.