A data-driven graph query method and apparatus

Abstract

This invention discloses a data-driven graph query method and apparatus, belonging to the field of graph computation efficiency optimization. The method includes: grouping queries with high similarity from a query pool into the same query set to obtain multiple query sets; dividing the graph structure to be queried into multiple shared graph tiles; sequentially executing each query set, performing point-to-point queries in parallel for the currently executing query set, and calculating the priority of each shared graph tile in real time during execution, loading the shared graph tiles into the last-level cache in descending order of priority, thereby achieving data sharing; during query execution, classifying the vertices in the shared graph tiles to determine global vertices and core subgraphs, determining the initial boundary values ​​of the query based on the query results of global vertices, and updating the query path based on the query results of core subgraphs to obtain the optimal path. This invention can improve the system's execution efficiency for concurrent point-to-point queries of graph data.

Description

Technical Field

[0001] This invention belongs to the field of graph computation efficiency optimization, and more specifically, relates to a data-driven graph query method and apparatus. Background Technology

[0002] With the development of big data technology, graph data has been widely used. For example, in social networks, recommendation systems, and Internet of Things (IoT) networks, this data is typically represented in graph form. Taking social networks as an example, each social network user can be considered a node in the graph. These nodes can contain users' personal information, such as username, avatar, age, gender, and other attributes. Social relationships between users (such as friend relationships and following relationships) can be represented by edges. For example, if user A follows user B, there will be an edge in the graph pointing from A to B. Friend relationships between users change over time; new friend relationships create new edges, and the termination of existing relationships results in the deletion of edges.

[0003] In applications using graph-structured data, point-to-point queries are frequently involved, aiming to obtain a specific path between a source vertex and a destination vertex. This unique traversal characteristic significantly improves query efficiency and expands the applicability of point-to-point queries in everyday scenarios. Examples include optimizing logistics routes on maps, analyzing friend recommendations through social networks on social platforms, and analyzing risk propagation in financial assessments on payment platforms. The growing demands of these applications pose challenges to concurrent point-to-point queries on the underlying graph structure.

[0004] Previous research has followed a concurrent execution model for all point-to-point algorithms, neglecting the optimization potential of point-to-point algorithms. Even when optimizing point-to-point algorithms, the focus has been on improving the speed of individual point-to-point queries, ignoring the throughput of concurrent queries. Overall, existing concurrent point-to-point queries on graph data are inefficient and cannot effectively meet the needs of upper-layer applications. Summary of the Invention

[0005] In view of the shortcomings of the existing technology and the need for improvement, the present invention provides a data-driven graph query method and apparatus, the purpose of which is to improve the system's execution efficiency for concurrent point-to-point queries of graph data.

[0006] To achieve the above objectives, according to one aspect of the present invention, a data-driven graph query method is provided, comprising:

[0007] The query pool, consisting of peer-to-peer queries to be executed, is divided into multiple query sets. The similarity of peer-to-peer queries within the same query set is higher than the similarity of peer-to-peer queries between different query sets, and the size of the query set does not exceed the preset number of concurrent queries N.

[0008] Divide the graph structure to be queried into multiple shared graph tiles;

[0009] Each query set is executed sequentially; for the currently executing query set, the point-to-point queries within it are executed in parallel, and during the execution process, the priority of each shared tile is calculated in real time, and the shared tiles are loaded into the last-level cache in order of priority from high to low.

[0010] At any given time, the priority of a shared tile is positively correlated with the number of peer-to-peer queries it is associated with; the shared tiles associated with a peer-to-peer query contain the active vertices required by that peer-to-peer query.

[0011] Furthermore, for any two point-to-point queries q i and q j Their similarities are:

[0012]

[0013] Among them, S ij Represents a point-to-point query q i and q j Similarity between them; and These represent point-to-point queries q. i The source vertex and the destination vertex, and These represent point-to-point queries q. j The source vertex and the destination vertex; express and The distance between, express and The distance between them.

[0014] Furthermore, at any given time, the priority of shared tiles is:

[0015]

[0016] Where P represents the priority of the shared tile; N query This indicates the number of point-to-point queries associated with the shared tile; This indicates the proportion of hot vertices among the active vertices contained in the shared tile. Hot vertices are vertices whose degree exceeds a preset threshold; ω1 and ω2 are preset weight coefficients.

[0017] Furthermore, the edges of the graph structure to be queried are stored in the edge list in ascending order of source vertex ID;

[0018] Furthermore, the graph structure to be queried is divided into multiple shared graph tiles, including:

[0019] Based on the preset shared tile size, calculate the number of edges C that each shared tile can accommodate, and divide the edges in the edge list into multiple logical partitions in sequence; each logical partition contains C edges.

[0020] Store the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge into the corresponding data structure T. block middle;

[0021] Each logical partition corresponds to a shared graph structure.

[0022] Furthermore, the size of the shared tile satisfies the following condition: after removing the reserved redundant space of the last-level cache, the remaining space of the last-level cache is sufficient to store the shared tile and its corresponding logical partition information.

[0023] Furthermore, the shared tile size satisfies:

[0024]

[0025] Among them, S block S represents the size of the shared tile. graph V represents the size of the logical partition, |V| represents the total number of vertices, and V vertex S represents the storage cost of a single vertex. LLC S represents the size of the last-level cache. reserve This indicates that redundant space is reserved.

[0026] Furthermore, the data-driven graph query method provided by this invention further includes: performing a preprocessing operation when a shared graph tile is loaded into the last-level cache; the preprocessing operation includes:

[0027] Identify hot vertices in the shared graph and identify the M vertices with the highest degree among the hot vertices as global vertices, and use the graph structure corresponding to the remaining hot vertices as the core subgraph; hot vertices are vertices whose degree exceeds a preset threshold.

[0028] Obtain the query results from each global vertex to each vertex in the shared tile, and construct a query result table using the query results. Obtain the query results from each vertex in the shared graph to each global node, and construct a query result table using the query results. and In a point-to-point query where the source vertex and the destination vertex are the same, the query result is infinity;

[0029] Furthermore, for any vertex v s As the source vertex, with vertex v d Point-to-point query for the destination vertex Q(v) s ,v dThe initial boundary value (bound) is determined in the following ways:

[0030] From the query results table Obtain the vertex v s The query result R(v) of a point-to-point query with source vertex and destination vertex as global vertex. s ,v i );v i ∈V global V global Represents the global set of vertices;

[0031] From the query results table The process is to obtain the source vertex from the global vertex and the vertex v. d The query result R(v) of a point-to-point query for the destination vertex i ,v d );

[0032] For the global vertex set V global Each global vertex v in i Calculate R(v) s ,v i )+R(v i ,v d ), and initialize the minimum computation result as a point-to-point query Q(v s ,v d The boundary value of ).

[0033] Furthermore, the preprocessing operations also include:

[0034] Initialize query result table T subgraph This is used to record the query results between each pair of vertices in the core subgraph; in the core subgraph, the path between two hot vertex pairs is called a hot path segment.

[0035] Furthermore, point-to-point query Q(v) s ,v d The execution includes:

[0036] Step S1: Traverse vertex v s out-neighbor vertex v j And calculate vertex v s Its outgoing neighbor node v j The edge weight between vertices v is used as the edge weight between vertices v. s to v j The query result R between them, and v j As active vertices, we obtain the set of active vertices;

[0037] Step S2: If v s to v jIf it constitutes a valid path and R < bound, then update the value of the boundary value bound to the value of R; the valid path is a path with v s as the source vertex and v d as the destination vertex;

[0038] Step S3: If the active vertex set is empty, then go to step S7; otherwise, go to step S4;

[0039] Step S4: For each vertex v a in the active vertex set, if it is a hot vertex, then take each hot vertex as its out-neighbor vertex, and use the corresponding hot path segment query result in the query result table T subgraph as the edge weight; traverse the out-neighbor vertex v a of vertex v an , and calculate the edge weight W(v a , v an ) between vertex v a and its out-neighbor vertices v an , and update the query result between vertex v a and v an according to R = R + W(v s , v an );

[0040] Step S5: If R > bound, then prune the path between vertex v s and v an ; if there is a valid path in the remaining paths, and the minimum query result corresponding to the valid path is less than the boundary value bound, then update the boundary value to the minimum query result corresponding to the valid path;

[0041] Step S6: Take the destination vertex of the remaining paths as the active vertices of the next iteration, obtain the active vertex set of the next iteration, and go to step S3;

[0042] Step S7: Determine the valid path with the minimum query result in the remaining paths as the query path of the point-to-point query Q(v s , v d ), and determine the minimum query result as the final query result;

[0043] If the query path of the point-to-point query Q(v s , v d ) includes a hot path segment, and the query result table T subgraph does not record the query result of this hot path segment, then extract the query result of this hot path segment from the query path of the point-to-point query Q(v s , v d ) and record it in the query result table T subgraph .

[0044] According to another aspect of the present invention, a data-driven graph query apparatus is provided, comprising:

[0045] Computer-readable storage media and processors;

[0046] Computer-readable storage media are used to store computer programs;

[0047] The processor is used to read the computer program stored in the computer-readable storage medium and execute the data-driven graph query method provided by the present invention.

[0048] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0049] (1) This invention discovers that queries on the underlying graph structure will have an overlap effect, that is, 72%-90% of the vertices in the graph are accessed by multiple queries, which will lead to a large amount of redundant data access and seriously affect the execution efficiency of concurrent point-to-point queries. Based on this discovery, this invention divides point-to-point queries with high similarity into the same query set, and the point-to-point queries will be executed in the same batch. Similar point-to-point queries have a high degree of data overlap during path traversal, so point-to-point queries in the same batch can share the same data blocks, realizing data sharing. This can avoid repeatedly loading the same data into the last-level cache (LLC). At the same time, this invention divides the graph structure into fine-grained shared graph tiles and determines the priority of loading each shared graph tile into the LLC according to the number of associated queries, so that shared graph tiles with more associated queries will be loaded into the LLC first, which can effectively improve the hit rate of LLC during query execution, thereby effectively improving the execution efficiency of concurrent point-to-point queries.

[0050] (2) Point-to-point query involves path query, and the source vertex and destination vertex of the query are also the source vertex and destination vertex of the queried path. In the preferred embodiment of the present invention, for two point-to-point queries, the similarity is calculated based on the distance between the source vertex and the destination vertex. Specifically, the reciprocal of the sum of the distance between the source vertex and the distance between the destination vertex is used as the similarity between the queries, which can accurately calculate the similarity between the queries.

[0051] (3) Considering that hot vertices have a high access frequency, in the preferred embodiment of the present invention, the weighted sum of the number of associated queries and the proportion of hot vertices in the active vertices is used as the priority of the corresponding shared map pieces. This makes the shared map pieces with more associated queries and a higher proportion of hot vertices have a higher priority and be loaded into LLC in a limited way. This can further improve the hit rate of LLC and reduce the number of cache replacements, thereby further improving the execution efficiency of concurrent point-to-point queries.

[0052] (4) In a preferred embodiment of the present invention, physically continuous graph structure data is divided into logically independent blocks by means of logical partitioning, and based on the ordered nature of the edge list, only the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge are stored in the corresponding data structure T. block In this way, while ensuring the completion of the shared tile division, the maintenance overhead of the shared tiles is reduced.

[0053] (5) In a preferred embodiment of the present invention, the determined size of the shared tile ensures that a shared tile and related information can be completely stored in the LLC, thereby making full use of the LLC capacity; in a further preferred embodiment, the maximum granularity of the shared tile is determined by considering factors such as the logical partition size, the total number of vertices, the average storage cost of vertices, the number of concurrent queries, the LLC size, and the reserved redundant space, so as to adapt to the LLC size while avoiding excessive scheduling overhead.

[0054] (6) In a preferred embodiment of the present invention, when performing point-to-point query, hot vertices in the shared map are identified based on their degree, and the hot vertices with the highest degree are taken as global vertices. The initial boundary value of the query is determined by the path passing through the global vertices, thereby optimizing the pruning mechanism. When the point-to-point query is started, paths that are unlikely to become the final query path can be effectively eliminated, avoiding unnecessary calculations, reducing computational overhead and improving query efficiency.

[0055] (7) The present invention has found that different queries may involve repeated calculations of the same path segment, which leads to redundant calculations. Based on this finding, in the preferred embodiment of the present invention, hot vertices other than global vertices are assigned to the core subgraph. The path segment between two vertices in the core subgraph is defined as a hot path segment, and the query result of the hot path segment is recorded. When performing point-to-point queries, the query result of the hot path segment will only be calculated once, and subsequent queries will directly use the calculation result, thereby realizing computation sharing. Based on computation sharing, the present invention can effectively reduce redundant calculations, improve query efficiency, and reduce computational overhead. Attached Figure Description

[0056] Figure 1 Here is an example of performing a point-to-point query using existing methods; where (a) is an example of a graph structure and (b) is a schematic diagram of the execution process of a point-to-point query Q(v1,v9);

[0057] Figure 2 This is a schematic diagram illustrating data redundancy between concurrent queries provided by the present invention;

[0058] Figure 3 This is a schematic diagram illustrating computational redundancy between concurrent queries provided by the present invention;

[0059] Figure 4 This is a data-driven graph query method provided in Embodiment 1 of the present invention;

[0060] Figure 5 This is a schematic diagram of the identification of global vertices and core subgraphs and the construction of related query results provided in Embodiment 1 of the present invention; wherein, (a) is an example of graph structure, (b) is the representation of query results related to core vertices, and (c) is the representation of query results related to core subgraphs. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0062] In this invention, the terms "first," "second," etc. (if present) in the invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0063] Point-to-point queries are used to retrieve specific query results between specific pairs of vertices in a graph, and are generally divided into two categories:

[0064] 1) Weighted graph algorithms, including point-to-point shortest path (PPSP), point-to-point maximum path (PPWP), and point-to-point minimum path (PPNP), are used to identify the shortest, widest, or narrowest path between two vertices, respectively. Weighted graph algorithms have been widely used in social / financial networks, traffic planning, monitoring money laundering activities, and network quality analysis.

[0065] 2) Unweighted graph algorithms, including Breadth-First Search (BFS), Connectivity Search, and Reachability Search, are used to determine the connectivity and reachability between specific layers in an undirected graph and between specific pairs of points in a directed graph, respectively. Unweighted graph algorithms are widely used in advanced algorithms for biconnectivity, higher-order connectivity, and graph clustering.

[0066] Point-to-point queries involve path traversal, as follows: Figure 1 Taking PPSP as an example, we will explain the process of performing point-to-point queries. Figure 1 Figure (a) shows the structural details of the diagram, v1~v 12 Let v1 be the source vertex and v9 be the destination vertex. The corresponding edge weight is W(v1, v9), and the query is Q(v1, v9). Figure 1As shown in (b), when query Q(v1, v9) is executed, the specific details of the traversal path are included, including the determination of the active vertex (v i ), and the query result R(v1, v i ) from v1 to vi in each iteration. The query result represents the shortest distance between the corresponding vertices, and its value is dynamically updated during the iteration, and its interpretation varies according to the specific algorithm.

[0067] In each iteration, the optimal query result from the source vertex to the target vertex is defined as the boundary value, which is continuously updated during the iteration.

[0068] At the initial moment, v1 is the only active vertex. The query result is denoted as R(v1, v1) and set to 0, and the query results of other vertices are initialized to infinity. Subsequent iterations involve traversing the out-neighbor vertices (v i ) of the active vertex. If R(v1, v j ) + W(v i , v i ) < bound or R(v1, v j ) + W(v i , v i ) < bound = ∞, then R(v1, v j ) is updated to the value of R(v1, v j ) + W(v i , v i , v j ), and v j becomes the active vertex in the next iteration. Otherwise, the corresponding path will be pruned.

[0069] In the first iteration, traverse the out-neighbor vertices (v2 and v6) of the source vertex (v1), and obtain the query results R(v1, v2) from v1 to v2 and R(v1, v6) from v1 to v6, which are the weights of the edges between v2 and v6 and the source vertex v1 respectively. v2 and v6 are activated and added to the set of active vertices (V active ) in the next iteration. In the second iteration, an effective path from the source vertex to the target vertex is identified: v1 → v6 → v9. The query result R(v1, v9) of this path is less than the current boundary value, so the boundary value is updated to R(v1, v9). All paths with query results greater than the boundary value will be terminated in advance, resulting in the pruning of the paths to v3, v7 and v 11 . As the iteration progresses, the boundary will be continuously updated. In the fourth iteration, a better query path is found: v1 → v2 → v5 → v 10→v9, the query results for this path will be used to update the boundary values. At this point, all output edges of the active vertex have been pruned, resulting in no active vertices in the next iteration. Therefore, the query terminates early, and the boundary values ​​at this point become the final query results. During the traversal, the boundary value-based pruning step effectively reduces redundant data access and computation in point-to-point queries.

[0070] For other point-to-point queries, a similar path traversal process will be performed, which will not be listed here. Concurrent point-to-point queries involve batching multiple different queries together, with point-to-point queries within the same batch being executed concurrently. Existing graph query methods limit the system's execution efficiency for concurrent point-to-point queries and also affect the performance of practical applications. To address this issue, this invention provides an in-depth analysis of the relevant influencing factors.

[0071] In real-world graphs, a small subset of vertices connects to the majority of edges, forming a characteristic power-law distribution. This invention refers to these highly interconnected vertices as "hot vertices," and the convergent path obtained by performing point-to-point queries on pairs of hot vertices is called a "hot path." Through in-depth analysis of the concurrent query process, hot vertices and hot paths account for the majority of redundant overhead. Figure 1 For example, execute queries Q(v1,v9) and Q(v1,v9)... 10 When dealing with Q(v2, v9) and Q(v2, v9), according to existing graph computation methods, the process from v2 to v9 will be recalculated independently. 10 This path results in redundant overhead. Based on the definition of this invention, v5 and v 10 The vertex is identified as a hot vertex, path v2→v5→v 10 This is known as a hot path.

[0072] Specifically, during the traversal, queries originating from different vertices often visit the same vertices. Figure 2 The traversal paths of concurrent queries in the same batch were recorded, and the proportion of vertices visited multiple times by different queries to the total number of activated vertices was analyzed. Figure 2 The results show that when the query concurrency in batch processing reaches 512, over 90% of vertices are accessed multiple times. However, in the existing task-driven execution module, different queries access their required data independently, leading to redundant access to the same data. Allowing queries to share access to the same graph structure data can amortize data access overhead and significantly improve the overall throughput of concurrent query processing.

[0073] Furthermore, during the traversal process, various queries frequently recalculate the results of the same hot path segments, leading to redundant calculations. Figure 3The intersections of various query paths were recorded, leading to significant redundant computations in batch processing. For a specific graph structure, selecting the top 522 vertices (less than 1% of the total) with the highest degree as hot vertices and analyzing the proportion of these hot vertices in the query intersections at different concurrency levels revealed that as concurrency increases, the total amount of overlapping data decreases, while the percentage of hot vertices in this overlap can increase to 95% (at 512 concurrency). This indicates that hot vertices and their corresponding hot paths significantly impact redundant computations. Facilitating the sharing of hot path computations between different queries can significantly reduce computational costs and improve query efficiency.

[0074] The following are relevant examples.

[0075] Example 1:

[0076] A data-driven graph query method, such as Figure 4 As shown, it includes:

[0077] The query pool, consisting of peer-to-peer queries to be executed, is divided into multiple query sets. The similarity of peer-to-peer queries within the same query set is higher than the similarity of peer-to-peer queries between different query sets, and the size of the query set does not exceed the preset number of concurrent queries N.

[0078] Divide the graph structure to be queried into multiple shared graph tiles;

[0079] Each query set is executed sequentially; for the currently executing query set, the point-to-point queries within it are executed in parallel, and during the execution process, the priority of each shared tile is calculated in real time, and the shared tiles are loaded into the last-level cache in order of priority from high to low.

[0080] At any given time, the priority of a shared tile is positively correlated with the number of peer-to-peer queries it is associated with; the shared tiles associated with a peer-to-peer query contain the active vertices required by that peer-to-peer query.

[0081] Studies have shown that similar queries access data with a high degree of overlap. In this embodiment, point-to-point queries with high similarity in the query pool are divided into the same batch, so that queries in the same batch can share the same data block, thus realizing data sharing. This avoids repeatedly loading the same data into the last-level cache (LLC).

[0082] Since point-to-point queries involve path lookups, and the source and destination vertices of the query are also the source and destination vertices of the queried path, in this preferred embodiment, for any two point-to-point queries q... i and q j Their similarities are:

[0083]

[0084] Among them, S ij Represents a point-to-point query q i and q j Similarity between them; and These represent point-to-point queries q. i The source vertex and the destination vertex, and These represent point-to-point queries q. j The source vertex and the destination vertex; express and The distance between, express and The distance between vertices. Vertices are typically stored in memory as vectors; therefore, in practical applications, the distance between vertices can be calculated using Euclidean distance or similar methods.

[0085] The more queries associated with a shared graph tile, the less data redundancy will be reduced after the tile is shared. Furthermore, due to the locality of data access, data with higher access frequency is accessed more frequently; loading it into an LLC can effectively reduce LLC misses. Therefore, to maximize the benefits of data sharing, as a preferred implementation, this embodiment considers both the number of queries associated with the shared graph tile and the hot vertices it contains when calculating the priority of a shared graph tile. Specifically, at any given time, the priority of a shared graph tile is:

[0086]

[0087] Where P represents the priority of the shared tile; N query This indicates the number of point-to-point queries associated with the shared tile; This represents the proportion of hot vertices among the active vertices contained in the shared tile. Hot vertices are vertices whose degree exceeds a preset threshold. ω1 and ω2 are preset weight coefficients. Optionally, in this embodiment, ω1 is set to 0.5 and ω2 is set to 1.

[0088] Caching plays a crucial role in tiered storage systems, and determining the appropriate size of shared tiles is essential for maximizing cache utilization. Ideally, the size of shared tiles should be small enough to accommodate LLC (Limited Chain Cache), but not too small; otherwise, frequent cache replacements will be required during query processing, resulting in excessive scheduling overhead.

[0089] Typically, the edges of the graph structure being queried are stored in an edge list in ascending order of source vertex IDs, and each edge occupies a fixed size. Therefore, after determining the size of the shared graph tile, the maximum number of edges that the shared tile can accommodate is calculated. Subsequently, the edges in the edge list can be partitioned according to the calculated edge count. Since the edge list is sorted in ascending order of source vertex IDs, recording the start and end vertex IDs of each tile is sufficient. Because each vertex has a different out-degree, uniform partitioning may distribute the different output edges of vertices across multiple tiles. Therefore, we additionally store the count of edges associated with a specific vertex in a given tile. This method allows us to partition physically contiguous graph structure data into logically independent tiles. Partition information is stored in T. block In the process of data access, we can refer to T block The physical location of a block is determined by the vertex ID and the number of edges stored in the database.

[0090] Based on this, in this embodiment, the graph structure to be queried is divided into multiple shared graph tiles, specifically including:

[0091] Based on the preset shared tile size, calculate the number of edges C that each shared tile can accommodate, and divide the edges in the edge list into multiple logical partitions in sequence; each logical partition contains C edges.

[0092] Store the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge into the corresponding data structure T. block middle;

[0093] Each logical partition corresponds to a shared graph structure.

[0094] In this embodiment, the size of the shared tile satisfies the following condition: after removing the reserved redundant space of the last-level cache, the remaining space of the last-level cache is sufficient to store the shared tile and its corresponding logical partition information.

[0095] Specifically, this embodiment considers factors such as logical partition size, total number of vertices, average storage cost of vertices, number of concurrent queries, LLC size, and reserved redundant space to determine the maximum granularity of shared tiles, thus adapting to the LLC size while avoiding excessive scheduling overhead. The specific shared tile size satisfies:

[0096]

[0097] Among them, S block S represents the size of the shared tile. graph V represents the size of the logical partition, |V| represents the total number of vertices, and V vertex S represents the storage cost of a single vertex. LLC S represents the size of the last-level cache. reserveThis indicates that redundant space is reserved. Based on the shared tile size determined in the above manner, by considering the graph structure and querying specific information, the maximum granularity of each shared tile can be determined, ensuring effective utilization of available LLC capacity.

[0098] To further improve the execution efficiency of concurrent point-to-point queries, this embodiment proposes a two-layer computation sharing mechanism based on the computational redundancy associated with hot vertices and hot path segments, which shares query results among different queries.

[0099] To identify hot vertices and hot path segments, this embodiment further includes: performing a preprocessing operation when a shared tile is loaded into the last-level cache; the preprocessing operation includes:

[0100] Identify hot vertices in the shared graph and identify the M vertices with the highest degree among the hot vertices as global vertices. The graph structures corresponding to the remaining hot vertices are used as core subgraphs. Hot vertices are vertices with a degree exceeding a preset threshold, where M is a positive integer. The set of global vertices is denoted as V. global The set of the remaining hot vertices is denoted as V. subgraph ;

[0101] By using a point-to-all query algorithm, the query results from each global vertex to each vertex in the shared tile are obtained, and a query result table is constructed using these results.

[0102] Using a point-to-all query algorithm, query results from each vertex in the shared graph to each global node are obtained, and a query result table is constructed using these results.

[0103] and In the graph, a point-to-point query where the source vertex and destination vertex are the same will result in infinity. In practical applications, this can be set to a value greater than the maximum path length in the graph.

[0104] Initialize query result table T subgraph This is used to record the query results between each pair of vertices in the core subgraph; in the core subgraph, the path between two hot vertex pairs is called a hot path segment.

[0105] and T subgraph In this table, row and column indexes correspond to the source and destination vertices of the query. To maintain lightweight operation, unlike the query result table corresponding to global vertices, T... subgraph In the preprocessing stage, only null values ​​are created and initialized, without direct computation. This embodiment further finds that if a convergent query path is obtained between two vertices, then any path segment extracted from that path also satisfies the convergence condition. Therefore, Tsubgraph The query results will be gradually extracted and populated into the table during subsequent queries.

[0106] by Figure 5 For example, (a) shows a specific graph structure. Based on degree, vertex v6 ​​is identified as a global vertex, and vertices v2, v3, v5, v7, and v6 are identified as global vertices. 10 Assigned to the core subgraph; for global vertices, a query result table can be constructed. and like Figure 5 As shown in (b); for the core subgraph, initialize the query result table T. subgraph ,like Figure 5 As shown in (c) in the figure.

[0107] Since the global vertex has the highest degree, each query path is highly likely to pass through the global vertex. In this embodiment, the query result table established for the global vertex will be used to determine the initial boundary values ​​for point-to-point queries, thereby effectively pruning at the initial moment of the query and avoiding unnecessary computation, thus achieving the first layer of computational sharing. Specifically, for any vertex v... s As the source vertex, with vertex v d Point-to-point query for the destination vertex Q(v) s ,v d The initial boundary value (bound) is determined in the following ways:

[0108] From the query results table Obtain the vertex v s The query result R(v) of a point-to-point query with source vertex and destination vertex as global vertex. s ,v i (i.e., obtaining the query results table) Mid-vertex v s The query results for the corresponding row; v i ∈V global V global Represents the global set of vertices;

[0109] From the query results table The process is to obtain the source vertex from the global vertex and the vertex v. d The query result R(v) of a point-to-point query for the destination vertex i ,v d (i.e., obtaining the query results table) vertex v in d The query results for the corresponding column;

[0110] For the global vertex set V global Each global vertex v in i Calculate R(v) s,v i )+R(v i ,v d ), and initialize the minimum computation result as a point-to-point query Q(v s ,v d The boundary value of ).

[0111] It's easy to understand that the boundary value `bound` will also be continuously updated in subsequent iterations.

[0112] As mentioned earlier, hot path segments are frequently recalculated for different queries. This embodiment utilizes the corresponding query result table T of the core subgraph. subgraph The query results for these hot path segments are recorded and shared for query purposes, thus achieving a second layer of computational sharing. During path traversal, if the active vertex being traversed is a hot vertex v... h Then you can go to T subgraph The query results for the corresponding row are found in the table, and that row records the hot vertex v. h For query results to other hot vertices, if a query result is not empty, it indicates that a previous query task traversed the corresponding hot path segment, obtained the result, and then processed it in T. subgraph The relevant query results are stored in the database. At this point, the hot vertex corresponding to this non-empty query result is considered as the hot vertex v. h The outgoing neighbor vertices are directly used as hot path segments, and the query results of the corresponding hot path segments are regarded as edge weights. They are used together with the regular outgoing neighbor vertices in the calculation and update of the active vertex set. This reduces redundant calculation of hot path segments and speeds up traversal.

[0113] With the help of T subgraph This embodiment is achieved through... and Determine point-to-point query Q(v) s ,v d After the boundary value bound is obtained, a point-to-point query is performed on Q(v). s ,v d The execution includes:

[0114] Step S1: Traverse vertex v s out-neighbor vertex v j And calculate vertex v s Its outgoing neighbor node v j The edge weight between vertices v is used as the edge weight between vertices v. s to v j The query result R between them, and v j As active vertices, we obtain the set of active vertices;

[0115] Step S2: If v s to v jIf it constitutes a valid path and R < bound, then update the value of the boundary value bound to the value of R; the valid path is a path with v s as the source vertex and v d as the destination vertex;

[0116] Step S3: If the active vertex set is empty, then go to step S7; otherwise, go to step S4;

[0117] Step S4: For each vertex v a in the active vertex set, if it is a hot vertex, then take each hot vertex as its out-neighbor vertex, and use the corresponding hot path segment query result in the query result table T subgraph as the edge weight; traverse the out-neighbor vertex v[[ID=​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​Since any path segment of a convergent path is itself convergent, taking PPSP as an example, if the shortest path between two vertices is found, then any path segment on that path is also the shortest path between the corresponding two vertices. In practical applications, if the known optimal query path contains multiple hot vertices, it means that there exists at least one convergent path segment consisting of two hot vertices at both ends. Optionally, in this embodiment, path segment information can be extracted in the following way:

[0123] When the query path passes through vertex v i At that time, record path v s →v i The query results and v on that path i The parent node, and this information will be retained until a new, better path reactivates v. i In this way, when the final query converges, the query results and parent nodes of each relevant vertex can be obtained.

[0124] Based on the recorded results, from the destination vertex v d Start by following the convergence path from the source vertex v s to v d Extract the query results of the source vertex and its corresponding parent vertex; then, using the obtained parent vertex, repeat the previous step to obtain the vertex closer to the source vertex v. s The process continues until the query results for each path segment are obtained. Once the query results for all path segments have been extracted, the result will be displayed in T. subgraph Update the query results at the corresponding positions in the table.

[0125] It should be noted that, due to the effective pruning operation used in point-to-point queries, a large amount of invalid computation is effectively avoided and the proportion of activated vertices is very small. Therefore, the extraction process of hot path query results can be completed with extremely low time and space overhead.

[0126] In summary, this embodiment proposes a data-driven graph query method to address the issue of overlapping traversal paths in different peer-to-peer queries. This method facilitates shared access to the same graph structure across different queries, avoids data redundancy, and distributes data access costs. Furthermore, to address the problem of frequently accessed path segments being recalculated by numerous queries, a two-layer computation sharing mechanism is proposed. This mechanism accelerates query convergence by sharing computed values ​​from frequently accessed paths across different queries, further improving the execution efficiency of concurrent peer-to-peer queries. Experiments show that the graph computation method provided in this embodiment outperforms the state-of-the-art peer-to-peer query system SGraph by an average of 3.2 times.

[0127] The following are some examples of the data-driven graph query method provided by the embodiments of the present invention in practical applications:

[0128] Application Example 1: Optimizing logistics routes on a map:

[0129] A graph structure data corresponding to the map is constructed, where graph vertices represent key locations in the logistics network, such as warehouses, distribution centers, and customer addresses, each with a unique identifier; edges represent logistics connections between vertices, which can be roads, railways, etc.; edge weights represent various costs and considerations when transporting goods, such as distance, time, and freight. During peak logistics periods, a large number of logistics demands are generated in a short period of time. The warehouses where these goods are stored and the user's destination correspond to the source and destination vertices of point-to-point queries, respectively. These demands together constitute the query pool, and the graph data is the previously constructed map, stored according to CSR and other methods. The path results are obtained through the above graph query method. If the map information does not change, the logistics route can continue to follow the path results.

[0130] Application Example 2: Recommending friends based on social network analysis on a social application:

[0131] This invention constructs a graph structure data corresponding to a social network, where vertices represent individual users in the social network; edges represent relationships between users, such as friendships and following relationships; and edge weights represent the strength or quality of social relationships, which can be based on various factors such as the number of mutual friends and interaction frequency. In practical applications, relying on the constructed graph structure data, other users with mutual friends with the target user are identified by analyzing the user's social network and recommended as potential friends. Social platforms can use the graph query method provided by this invention to quickly recommend a large number of friends, enabling new users to quickly build their own social circles.

[0132] Application Example 3: Analyzing the propagation of risk in financial assessment using a payment application:

[0133] A graph structure data structure corresponding to the payment application is constructed, where the vertices of the graph represent participants in the payment network, such as users, merchants, and financial institutions; edges represent financial transactions or relationships, such as a payment or transfer; and edge weights represent attributes such as transaction amount, transaction frequency, and transaction speed. The application in this scenario is primarily for real-time monitoring and analysis of fund flows and risk propagation within the payment network. The path obtained from the point-to-point query in the graph structure data is the transaction path. For certain merchants, the path with the highest transaction frequency or the largest transaction amount can be tracked to monitor for potential financial risks. Some nodes may generate multiple transactions simultaneously, making the method of this invention suitable for synchronous, fast, and accurate tracking.

[0134] Example 2:

[0135] A data-driven graph query apparatus includes: a computer-readable storage medium and a processor;

[0136] Computer-readable storage media are used to store computer programs;

[0137] The processor is used to read the computer program stored in the computer-readable storage medium and execute the data-driven graph query method provided in Embodiment 1 above.

[0138] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data-driven graph query method, characterized in that, include: The query pool, consisting of peer-to-peer queries to be executed, is divided into multiple query sets; The similarity of peer-to-peer queries within the same query set is higher than the similarity of peer-to-peer queries between different query sets, and the size of the query set does not exceed the preset number of concurrent queries. N ; Divide the graph structure to be queried into multiple shared graph tiles; Each query set is executed sequentially; for the currently executing query set, the point-to-point queries within it are executed in parallel, and during the execution process, the priority of each shared tile is calculated in real time, and the shared tiles are loaded into the last-level cache in order of priority from high to low. At any given time, the priority of a shared graph tile is positively correlated with the number of peer-to-peer queries associated with it; the shared graph tile associated with a peer-to-peer query contains the active vertices required by that peer-to-peer query; the edges of the graph structure to be queried are stored in the edge list in ascending order of source vertex ID. Furthermore, the graph structure to be queried is divided into multiple shared graph tiles, including: Calculate the number of edges that each shared tile can accommodate, based on the preset shared tile size. C The edges in the edge list are then divided into multiple logical partitions in sequence; each logical partition contains... C Edge; Store the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge into the corresponding data structure. T block middle; Each logical partition corresponds to a shared graph structure.

2. The data-driven graph query method as described in claim 1, characterized in that, For any two point-to-point queries q i and q j Their similarities are: in, S ij Indicates point-to-point query q i and q j Similarity between them; and These represent point-to-point queries. q i The source vertex and the destination vertex, and These represent point-to-point queries. q j The source vertex and the destination vertex; express and The distance between, express and The distance between them.

3. The data-driven graph query method as described in claim 1, characterized in that, At any given time, the priority of shared tiles is: in, P Indicates the priority of shared tiles; N query This indicates the number of point-to-point queries associated with the shared tile; This indicates the proportion of hot vertices among the active vertices contained in a shared tile. Hot vertices are vertices whose degree exceeds a preset threshold. ω 1 and ω 2 represents the preset weighting coefficient.

4. The data-driven graph query method as described in any one of claims 1 to 3, characterized in that, The size of the shared tile satisfies the following condition: after removing the reserved redundant space of the last-level cache, the remaining space of the last-level cache is sufficient to store the shared tile and its corresponding logical partition information.

5. The data-driven graph query method as described in claim 4, characterized in that, Shared tile size satisfies: in, S block Indicates the size of the shared tile. S graph Indicates the size of the logical partition, | V | represents the total number of vertices. This represents the storage overhead of a single vertex. S LLC Indicates the size of the last-level cache. S reserve This indicates that redundant space is reserved.

6. The data-driven graph query method as described in claim 5, characterized in that, Also includes: When shared tiles are loaded into the last-level cache, preprocessing operations are performed. The preprocessing operations include: Identify hot vertices in shared tiles and select the hot vertices with the highest degree. M A few vertices are identified as global vertices, and the graph structure corresponding to the remaining hot vertices is used as the core subgraph; hot vertices are those whose degree exceeds a preset threshold. M It is a positive integer; Obtain the query results from each global vertex to each vertex in the shared tile, and construct a query result table using the query results. ; Obtain the query results from each vertex in the shared tile to each global node, and construct a query result table using the query results. ; and In a point-to-point query where the source vertex and the destination vertex are the same, the query result is infinity. Furthermore, for any vertex v s As the source vertex, with the vertex v d Point-to-point query for the destination vertex Q( v s , v d ), its initial boundary values bound The methods for determining include: From the query results table Get the vertex v s The query results of a point-to-point query with the source vertex as the source vertex and the global vertex as the destination vertex. R ( v s , v i ); , Represents the global set of vertices; From the query results table Obtain the source vertex from the global vertex, and the vertex from the vertex. v d Query results of a point-to-point query for the destination vertex R ( v i , v d ); For the global vertex set Each global vertex in v i ,calculate R ( v s , v i )+ R ( v i , v d ), and initialize the minimum computation result as a point-to-point query Q( v s , v d The boundary value of ).

7. The data-driven graph query method as described in claim 6, characterized in that, The preprocessing operation also includes: Initialize the query result table T subgraph This is used to record the query results between each pair of vertices in the core subgraph; in the core subgraph, the path between two hot vertex pairs is called a hot path segment. Furthermore, point-to-point query Q( v s , v d The execution includes: Step S1: Traverse vertices v s out-neighbor vertex v j And calculate the vertices v s Its neighboring nodes v j The edge weight between vertices is used as the vertex weight. v s arrive v j The query result R between them, and will v j As active vertices, we obtain the set of active vertices; Step S2: If v s arrive v j A valid path is formed, and R < bound Then the boundary value bound The value is updated to R The value; the valid path is... v s As the source vertex, with v d The path to the destination vertex; Step S3: If the active vertex set is empty, proceed to step S7; otherwise, proceed to step S4. Step S4: For each vertex in the active vertex set v a If a vertex is a hot vertex, then each hot vertex is treated as its out-neighbor vertex, and the query result table is updated accordingly. T subgraph The corresponding hot path segment query results are used as edge weights; traverse the vertices. v a out-neighbor vertex v an And calculate the vertices v a Each of them originates from the neighboring vertices v an The edge weight W between them v a , v an ), and according to R=R+W( v a , v an Update vertices v s arrive v an Query results between; Step S5: If R> bound Then the vertex v s arrive v an Paths between nodes are pruned; if a valid path exists among the remaining paths, and the minimum query result corresponding to the valid path is less than the boundary value. bound If so, the boundary value is updated to the minimum query result corresponding to the valid path; Step S6: Use the destination vertex of the remaining path as the active vertex of the next iteration to obtain the active vertex set of the next iteration, and then proceed to step S3; Step S7: Determine the valid path with the smallest query result among the remaining paths as the point-to-point query Q( v s , v d The query path is determined, and the minimum query result is taken as the final query result. If point-to-point query Q( v s , v d The query path includes hot path segments, and the query result table T subgraph If the query results for this hot path segment are not recorded in the database, then a point-to-point query Q( v s , v d The query results for the hot path segment are extracted and recorded in the query results table. T subgraph middle.

8. A data-driven graph query device, characterized in that, include: Computer-readable storage media and processors; The computer-readable storage medium is used to store computer programs; The processor is used to read the computer program stored in the computer-readable storage medium and execute the data-driven graph query method according to any one of claims 1 to 7.

Images