Data-driven graph query method and device

Abstract

The invention discloses a data-driven graph query method and device, and belongs to the field of graph calculation efficiency optimizing.The data-driven graph query method comprises the steps that queries with high similarity in a query pool are divided into the same query set, and a plurality of query sets are obtained; dividing a to-be-queried graph structure into a plurality of shared graph blocks; all the query sets are executed in sequence, point-to-point query in the currently executed query set is executed in parallel, in the execution process, the priorities of all the shared image blocks are calculated in real time, the shared image blocks are loaded into a last-stage cache according to the sequence of the priorities from high to low, and therefore data sharing is achieved; when the query is executed, the vertexes in the shared image blocks are classified, global vertexes and core subgraphs are determined, the initial boundary value of the query is determined according to the query result of the global vertexes, and the query path is updated according to the query result of the core subgraphs, so that the optimal path is obtained. According to the method, the execution efficiency of a system on graph data concurrent point-to-point query can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of graph computing efficiency optimization, and more specifically, relates to a data-driven graph query method and device. Background Art

[0002] With the development of big data technology, graph data has been widely used. Such as social networks, recommendation systems, Internet of Things (IoT) networks, etc., these data are usually represented in the form of graphs. Taking social networks as an example, each social network user can be regarded as a node in the graph. These nodes can contain the user's personal information, such as username, avatar, age, gender and other attributes. The social relationships between users (such as friend relationships, follow relationships) can be represented by edges. For example, if user A follows user B, there will be an edge pointing from A to B in the graph. The friendship between users will change over time, new friendships will create new edges, and the severance of existing relationships will result in the deletion of edges.

[0003] In the application of graph structured data, point-to-point queries are often involved, the purpose of which is to obtain a specific path between the source vertex and the destination vertex. This unique traversal feature significantly improves query efficiency and expands the applicability of point-to-point queries in daily scenarios. For example, optimizing logistics routes on maps, recommending friends through social network analysis on social platforms, and analyzing risk propagation in financial assessments on payment platforms. The growing demands of these applications pose challenges to concurrent point-to-point queries on the underlying graph structure.

[0004] Previous studies have followed the point-to-point concurrent execution model of all algorithms, ignoring the optimization potential of point-to-point algorithms; even if point-to-point algorithms are optimized, they either focus on optimizing the speed of a single point-to-point query and ignore the throughput of concurrent queries. In general, the existing concurrent point-to-point query of graph data is not efficient and cannot effectively meet the needs of upper-level applications. Summary of the invention

[0005] In view of the defects and improvement needs of the prior art, the present invention provides a data-driven graph query method and device, which aims to improve the execution efficiency of the system for concurrent point-to-point queries on graph data.

[0006] To achieve the above object, according to one aspect of the present invention, a data-driven graph query method is provided, comprising:

[0007] A query pool consisting of point-to-point queries to be executed is divided into multiple query sets; the similarity of point-to-point queries in the same query set is higher than the similarity of point-to-point queries between different query sets, and the size of the query set does not exceed a preset number of concurrent queries N;

[0008] Divide the graph structure to be queried into multiple shared blocks;

[0009] Execute each query set in sequence; for the currently executed query set, execute the point-to-point queries in parallel, and during the execution process, calculate the priority of each shared tile in real time, and load the shared tiles into the last-level cache in order from high to low priority;

[0010] Among them, at any time, the priority of a shared tile is positively correlated with the number of point-to-point queries associated with it; the shared tile associated with a point-to-point query contains the active vertices required by the point-to-point query.

[0011] Furthermore, for any two point-to-point queries q i and q j , the similarity is:

[0012]

[0013] Among them, S ij Represents a point-to-point query q i and q j similarities between and They represent point-to-point queries q i The source and destination vertices of and They represent point-to-point queries q j The source and destination vertices of express and The distance between express and The distance between.

[0014] Furthermore, at any time, the priority of shared tiles is:

[0015]

[0016] Where P represents the priority of the shared tile; N query Indicates the number of peer-to-peer queries associated with the shared tile; It represents the proportion of hot vertices in the active vertices contained in the shared tile. Hot vertices are vertices whose degrees exceed the preset threshold. ω1 and ω2 are preset weight coefficients.

[0017] Furthermore, the edges of the graph structure to be queried are stored in the edge list in ascending order of the source vertex IDs;

[0018] In addition, the graph structure to be queried is divided into multiple shared blocks, including:

[0019] According to the preset shared tile size, the number of edges C that each shared tile can accommodate is calculated, and the edges in the edge list are divided into multiple logical partitions in order; each logical partition contains C edges;

[0020] Store the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge into the corresponding data structure T block middle;

[0021] The graph structure corresponding to each logical partition is a shared block.

[0022] Furthermore, the size of the shared tiles satisfies that after removing the reserved redundant space of the last-level cache, the remaining space of the last-level cache is sufficient to store the shared tiles and the corresponding logical partition information.

[0023] Furthermore, the shared tile size satisfies:

[0024]

[0025] Among them, S block Indicates the size of the shared tile, S graph represents the size of the logical partition, |V| represents the total number of vertices, and V vertex represents the storage cost of a single vertex, S LLC Indicates the size of the last-level cache, S reserve Indicates that redundant space is reserved.

[0026] Furthermore, the data-driven graph query method provided by the present invention further includes: when the shared image block is loaded into the last-level cache, performing a preprocessing operation; the preprocessing operation includes:

[0027] Identify the hot vertices in the shared block, and identify the M vertices with the highest degree among the hot vertices as global vertices, and the graph structures corresponding to the remaining hot vertices as core subgraphs; hot vertices are vertices whose degrees exceed a preset threshold;

[0028] Get the query results from each global vertex to each vertex in the shared block, and use the query results to build a query result table Get the query results from each vertex in the shared block to each global node respectively, and use the query results to build the query result table and In the point-to-point query, the query result is infinite if the source vertex and the destination vertex are the same;

[0029] And, for any vertex v s is the source vertex, with vertex v d Point-to-point query Q(v s ,v d), the initial boundary value bound is determined by:

[0030] From the query result table Get the vertex v s The query result R(v s ,v i );v i ∈V global , V global Represents the global vertex set;

[0031] From the query result table Get the global vertex as the source vertex and the vertex v d The query result R(v i ,v d );

[0032] For the global vertex set V global For each global vertex v in i , calculate R(v s ,v i )+R(v i ,v d ), and initialize the minimum calculation result as the point-to-point query Q(v s ,v d )’s boundary value is bound.

[0033] Furthermore, the preprocessing operation also includes:

[0034] Initialize query result table T subgraph , used to record the query results between each vertex pair in the core subgraph; in the core subgraph, the path between two hot vertex pairs is called a hot path segment;

[0035] And, the point-to-point query Q(v s ,v d ) includes:

[0036] Step S1: traverse vertex v s The outgoing neighbor vertex v j , and calculate the vertex v s and its neighbor nodes v j The edge weight between the vertices v s to v j The query result R between them, and v j As the active vertex, get the active vertex set;

[0037] Step S2: If v s to v jIf it constitutes a valid path and R < bound, then update the value of the boundary value bound to the value of R; the valid path is a path with v s as the source vertex and v d as the destination vertex;

[0038] Step S3: If the active vertex set is empty, then go to step S7; otherwise, go to step S4;

[0039] Step S4: For each vertex v a in the active vertex set, if it is a hot vertex, then take each hot vertex as its out-neighbor vertex, and use the corresponding hot path segment query result in the query result table T subgraph as the edge weight; traverse the out-neighbor vertex v a of vertex v an , and calculate the edge weight W(v a , v an ) between vertex v a and its out-neighbor vertices v an , and update the query result from vertex v a to v an according to R = R + W(v s , v an );

[0040] Step S5: If R > bound, then prune the path between vertex v s and v an ; if there is a valid path in the remaining paths, and the minimum query result corresponding to the valid path is less than the boundary value bound, then update the boundary value to the minimum query result corresponding to the valid path;

[0041] Step S6: Take the destination vertex of the remaining paths as the active vertices of the next iteration, obtain the active vertex set of the next iteration, and go to step S3;

[0042] Step S7: Determine the valid path with the minimum query result in the remaining paths as the query path of the point-to-point query Q(v s , v d ), and determine this minimum query result as the final query result;

[0043] If the query path of the point-to-point query Q(v s , v d ) includes a hot path segment, and the query result table T subgraph does not record the query result of this hot path segment, then extract the query result of this hot path segment from the query path of the point-to-point query Q(v s , v d ) and record it in the query result table T subgraph .

[0044] According to another aspect of the present invention, there is provided a data-driven graph query device, comprising:

[0045] A computer readable storage medium and a processor;

[0046] The computer readable storage medium is used to store the computer program;

[0047] The processor is used to read the computer program stored in the computer-readable storage medium and execute the above-mentioned data-driven graph query method provided by the present invention.

[0048] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0049] (1) The present invention finds that there will be an overlapping effect when querying the underlying graph structure, that is, 72%-90% of the vertices in the graph are accessed by multiple queries, which will lead to a large amount of redundant data access and seriously affect the execution efficiency of concurrent point-to-point queries. Based on this finding, the present invention divides point-to-point queries with high similarity into the same query set, where the point-to-point queries will be executed in the same batch. Similar point-to-point queries have a high degree of overlap in accessed data during the path traversal process, so that point-to-point queries in the same batch can share the same data blocks, thereby realizing data sharing, thereby avoiding repeatedly loading the same data into the last-level cache (LLC). At the same time, the present invention divides the graph structure into fine-grained shared blocks, and determines the priority of each shared block to be loaded into the LLC according to the number of associated queries, so that shared blocks with a large number of associated queries will be loaded into the LLC first, which can effectively improve the hit rate of hitting the LLC during query execution, thereby effectively improving the execution efficiency of concurrent point-to-point queries.

[0050] (2) Point-to-point query involves path query, and the source vertex and destination vertex of the query are also the source vertex and destination vertex of the queried path. In the preferred embodiment of the present invention, for two point-to-point queries, the similarity calculation is completed based on the distance between the source vertex and the destination vertex. Specifically, the reciprocal of the sum of the distance between the source vertices and the distance between the destination vertices is used as the similarity between the queries, which can accurately calculate the similarity between the queries.

[0051] (3) Considering that hot vertices have a higher access frequency, in the preferred embodiment of the present invention, a weighted sum of the number of associated queries and the proportion of hot vertices in the active vertices is used as the priority of the corresponding shared block. As a result, shared blocks that are associated with more queries and have a higher proportion of hot vertices will have a higher priority and will be loaded into the LLC in a limited manner. This can further improve the hit rate of the LLC, reduce the number of cache replacements, and further improve the execution efficiency of concurrent point-to-point queries.

[0052] (4) In the preferred embodiment of the present invention, physically continuous graph structure data is divided into logically independent blocks by means of logical partitions, and based on the order of the edge list, only the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge are stored in the corresponding data structure T block In this way, the maintenance overhead of shared tiles is reduced while ensuring the completion of shared tile division.

[0053] (5) In the preferred embodiment of the present invention, the determined shared tile size ensures that a shared tile and related information can be completely stored in the LLC, thereby fully utilizing the capacity of the LLC. In a further preferred embodiment, the maximum granularity of the shared tile is determined by taking into account factors such as the logical partition size, the total number of vertices, the average storage cost of vertices, the number of concurrent queries, the LLC size, and the reserved redundant space, thereby adapting to the LLC size while avoiding excessive scheduling overhead.

[0054] (6) In the preferred embodiment of the present invention, when executing a point-to-point query, hot vertices in the shared tile are identified based on the degree, and some of the hot vertices with the highest degree are used as global vertices. The initial boundary value of the query is determined with the help of the path passing through the global vertex, thereby optimizing the pruning mechanism. When the point-to-point query is started, the path that cannot become the final query path can be effectively eliminated, thereby avoiding unnecessary calculations, reducing the computational overhead and improving the query efficiency.

[0055] (7) The present invention finds that different queries may involve repeated calculations of the same path segment, which will lead to redundant calculations. Based on this finding, in a preferred embodiment of the present invention, the present invention assigns hot vertices other than global vertices to the core subgraph, and the path segment between two vertices in the core subgraph is defined as a hot path segment, and the query results of the hot path segment are recorded. When performing a point-to-point query, the query result of the hot path segment will only be calculated once, and subsequent queries will directly use the calculation result, thereby realizing calculation sharing. Based on calculation sharing, the present invention can effectively reduce redundant calculations, improve query efficiency and reduce calculation overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is an example of executing a point-to-point query using an existing method; wherein (a) is an example of a graph structure, and (b) is a schematic diagram of the execution process of a point-to-point query Q(v1,v9);

[0057] Figure 2 A schematic diagram of data redundancy between concurrent queries provided by the present invention;

[0058] Figure 3 A schematic diagram of computing redundancy between concurrent queries provided by the present invention;

[0059] Figure 4 The data-driven graph query method provided in Example 1 of the present invention;

[0060] Figure 5 A schematic diagram of the identification of global vertices, core subgraphs and the construction of related query results provided in Example 1 of the present invention; wherein (a) is an example of a graph structure, (b) represents the query results related to core vertices, and (c) represents the query results related to core subgraphs. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0062] In the present invention, the terms "first", "second", etc. (if any) in the present invention and the accompanying drawings are used to distinguish similar objects but not necessarily to describe a specific order or sequence.

[0063] Point-to-point queries are used to retrieve specific query results between specific pairs of vertices in the graph and are generally divided into two categories:

[0064] 1) Weighted graph algorithms, including point-to-point shortest path (PPSP), point-to-point maximum path (PPWP), and point-to-point minimum path (PPNP), which are used to identify the shortest, widest, or narrowest path between two vertices, respectively; weighted graph algorithms have been widely used in social / financial networks, traffic planning, monitoring money laundering activities, and network quality analysis.

[0065] 2) Unweighted graph algorithms, including breadth-first search (BFS), connectivity search, and reachability search, which are used to determine the connectivity and reachability of a specific layer in an undirected graph and a specific pair of points in a directed graph, respectively. Unweighted graph algorithms are widely used in advanced algorithms for biconnectivity, high-order connectivity, and graph clustering.

[0066] Point-to-point query involves path traversal. Figure 1 Take PPSP as an example to illustrate the process of performing point-to-point query on PPSP. Figure 1 (a) shows the structural details of the graph, v1~v 12 There are 12 vertices in the graph, and the numbers on the edges between vertices are edge weights. Take v1 as the source vertex and v9 as the destination vertex. The corresponding edge weight is W(v1, v9), and the query is Q(v1, v9). Figure 1As shown in (b) of, the specific details of the traversal path when executing the query Q(v1, v9) are included, including the determination of the active vertex (v i ), and the query result R(v1, v i ) from v1 to vi in each iteration. The query result represents the shortest distance between the corresponding vertices, and its value is dynamically updated during the iteration process, and its interpretation varies depending on the specific algorithm.

[0067] In each iteration, the optimal query result from the source vertex to the target vertex is defined as the boundary value, which will be continuously updated during the iteration process.

[0068] At the initial moment, v1 is the only active vertex. The query result is recorded as R(v1, v1) and set to 0, and the query results of other vertices are initialized to infinity. Subsequent iterations involve traversing the out-neighbor vertices (v i ) of the active vertex (v j ). If R(v1, v i ) + W(v i , v j ) < bound or R(v1, v i ) + W(v i , v j ) < bound = ∞, then R(v1, v j ) is updated to the value of R(v1, v i ) + W(v i , v j ), and v j becomes the active vertex in the next iteration. Otherwise, the corresponding path will be pruned.

[0069] In the first iteration, traverse the out-neighbor vertices (v2 and v6) of the source vertex (v1), and obtain the query results R(v1, v2) from v1 to v2 and R(v1, v6) from v1 to v6, which are the weights of the edges between v2 and v6 and the source vertex v1 respectively. v2 and v6 are activated and added to the set of active vertices (V active ) in the next iteration. In the second iteration, an effective path from the source vertex to the target vertex is identified: v1 → v6 → v9. The query result R(v1, v9) of this path is less than the current boundary value, so the boundary value is updated to R(v1, v9). All paths with query results greater than the boundary value will be terminated in advance, resulting in the pruning of the paths to v3, v7, and v 11 . As the iteration progresses, the boundary will be continuously updated. In the fourth iteration, a better query path is found: v1 → v2 → v5 → v 10→v9, the query result of this path will be used to update the boundary value. At this point, all outgoing edges of the active vertex have been pruned, resulting in no active vertex in the next iteration. Therefore, the query is terminated early, and the boundary value at this time becomes the final query result. During the traversal process, the pruning step based on the boundary value effectively reduces redundant data access and calculation in point-to-point queries.

[0070] For other point-to-point queries, a similar path traversal process will be performed, which will not be listed here one by one. Concurrent point-to-point query, that is, when multiple different queries are taken as a batch, the point-to-point queries in the same batch will be executed concurrently. The existing graph query method limits the execution efficiency of the system for concurrent point-to-point queries and also affects the performance of practical applications. In response to this problem, the present invention conducts an in-depth analysis of the relevant influencing factors.

[0071] In real-world graphs, a small number of vertices are connected to most edges, forming a characteristic power-law distribution. This paper refers to these highly interconnected vertices as hot vertices, and the convergence path obtained by performing point-to-point queries on hot vertex pairs is called a hot path. Through in-depth analysis of the concurrent query process, hot vertices and hot paths account for most of the redundant overhead. Figure 1 For example, execute the query Q(v1,v9), Q(v1,v 10 ) and Q(v2,v9), according to the existing graph calculation method, the calculation from v2 to v will be recalculated independently. 10 Based on the definition of the present invention, v5 and v 10 Identified as a hot vertex, the path v2→v5→v 10 It is called the hot path.

[0072] Specifically, during the traversal process, queries originating from different vertices often visit the same vertices. Figure 2 The traversal paths of concurrent queries in the same batch are recorded, and the proportion of vertices visited multiple times by different queries to the total activated vertices is analyzed. Figure 2 The results shown show that when the query concurrency in a batch reaches 512, more than 90% of the vertices are accessed multiple times. However, in the existing task-driven execution module, different queries access the data they need independently, resulting in redundant access to the same data. Allowing queries to share access to the same graph structure data can amortize the data access overhead and significantly improve the overall throughput of concurrent query processing.

[0073] Furthermore, during the traversal process, various queries often recalculate the results of the same hot path segments, resulting in redundant computations. Figure 3The intersections of various query paths are recorded, and these path intersections will lead to a large amount of redundant calculations in batch processing. For a specific graph structure data, when the first 522 (less than 1% of the total) vertices with the highest degree are selected as hot vertices, the proportion of these hot vertices in the query intersections at different concurrency levels is analyzed. It is found that as the concurrency increases, the total amount of overlapping data decreases, and the percentage of hot vertices in this overlap can increase to 95% (at 512 concurrency). This shows that hot vertices and their corresponding hot paths significantly affect redundant calculations. Promoting the sharing of hot path calculations between different queries can significantly reduce computing costs and improve query efficiency.

[0074] The following are relevant embodiments.

[0075] Embodiment 1:

[0076] A data-driven graph query method, such as Figure 4 As shown, including:

[0077] A query pool consisting of point-to-point queries to be executed is divided into multiple query sets; the similarity of point-to-point queries in the same query set is higher than the similarity of point-to-point queries between different query sets, and the size of the query set does not exceed a preset number of concurrent queries N;

[0078] Divide the graph structure to be queried into multiple shared blocks;

[0079] Execute each query set in sequence; for the currently executed query set, execute the point-to-point queries in parallel, and during the execution process, calculate the priority of each shared tile in real time, and load the shared tiles into the last-level cache in order from high to low priority;

[0080] Among them, at any time, the priority of a shared tile is positively correlated with the number of point-to-point queries associated with it; the shared tile associated with a point-to-point query contains the active vertices required by the point-to-point query.

[0081] Research shows that the data accessed by similar queries have a high degree of overlap. This embodiment divides point-to-point queries with high similarity in the query pool into the same batch, so that queries in the same batch can share the same data block, thereby realizing data sharing, thereby avoiding repeatedly loading the same data into the last-level cache (LLC).

[0082] Since the point-to-point query involves a path query, the source vertex and the destination vertex of the query are also the source vertex and the destination vertex of the queried path. Based on this, as a preferred implementation, in this embodiment, for any two point-to-point queries q i and q j , the similarity is:

[0083]

[0084] Among them, S ij Represents a point-to-point query q i and q j similarities between and They represent point-to-point queries q i The source and destination vertices of and They represent point-to-point queries q j The source and destination vertices of express and The distance between express and The distance between vertices. Usually, vertices are stored in the form of vectors in memory. Accordingly, in practical applications, the distance between vertices can be calculated by Euclidean distance, etc.

[0085] The more queries a shared block is associated with, the less data redundancy will be reduced after the shared block is shared. In addition, due to the locality of data access, the hotter the data is, the more frequently it is accessed. Loading it into LLC can effectively reduce the number of LLC misses. Based on this, in order to maximize the benefits of data sharing, as a preferred implementation, this embodiment will consider the number of queries associated with the shared block and the hot vertices contained therein when calculating the priority of the shared block. Specifically, at any time, the priority of the shared block is:

[0086]

[0087] Where P represents the priority of the shared tile; N query Indicates the number of peer-to-peer queries associated with the shared tile; Represents the proportion of hot vertices in the active vertices contained in the shared tile, and the hot vertices are vertices whose degrees exceed a preset threshold; ω1 and ω2 are preset weight coefficients, optionally, in this implementation, ω1 is set to 0.5 and ω2 is set to 1.

[0088] Cache plays a vital role in tiered storage systems, and properly determining the size of shared tiles is very important to maximize cache utilization. Ideally, the size of shared tiles should be small enough to fit in LLC, but not too small, otherwise frequent cache replacements will be required during query processing, causing excessive scheduling overhead.

[0089] Typically, the edges of the graph structure to be queried are stored in the edge list in ascending order of the source vertex ID, and each edge occupies a fixed size. Therefore, after determining the size of the shared block, the maximum number of edges that the shared block can accommodate is calculated, and then the edges in the edge list can be partitioned according to the calculated number of edges. Since the edge list is arranged in ascending order by the source vertex ID, it is sufficient to record the start and end vertex IDs of each block. Since the out-degree of each vertex is different, uniform partitioning may disperse different output edges of the vertex over multiple blocks. Therefore, we additionally store the count of edges associated with a specific vertex in a given block. This approach allows us to divide physically continuous graph structure data into logically independent blocks. Partition information is stored in T block In the data access process, we can refer to T block The vertex ID and edge number stored in determine the physical location of the block.

[0090] Based on this, in this embodiment, the graph structure to be queried is divided into multiple shared blocks, specifically including:

[0091] According to the preset shared tile size, the number of edges C that each shared tile can accommodate is calculated, and the edges in the edge list are divided into multiple logical partitions in order; each logical partition contains C edges;

[0092] Store the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge into the corresponding data structure T block middle;

[0093] The graph structure corresponding to each logical partition is a shared block.

[0094] In this embodiment, the size of the shared tiles satisfies that after removing the reserved redundant space of the last-level cache, the remaining space of the last-level cache is sufficient to store the shared tiles and the corresponding logical partition information.

[0095] Specifically, this embodiment considers the logical partition size, the total number of vertices, the average storage cost of vertices, the number of concurrent queries, the LLC size, and the reserved redundant space to determine the maximum granularity of the shared tiles, thereby avoiding excessive scheduling overhead while adapting to the LLC size. The shared tile size specifically meets the following requirements:

[0096]

[0097] Among them, S block Indicates the size of the shared tile, S graph represents the size of the logical partition, |V| represents the total number of vertices, and V vertex represents the storage cost of a single vertex, S LLC Indicates the size of the last-level cache, S reserveIndicates that redundant space is reserved. Based on the shared tile size determined in the above manner, the maximum granularity of each shared tile is determined by considering the graph structure and query-specific information, which can ensure the effective utilization of the available LLC capacity.

[0098] In order to further improve the execution efficiency of concurrent point-to-point queries, this embodiment proposes a two-layer computing sharing mechanism based on the computing redundancy associated with hot vertices and hot path segments to share their query results between different queries.

[0099] In order to identify hot vertices and hot path segments, the present embodiment further includes: when the shared tiles are loaded into the last level cache, performing a preprocessing operation; the preprocessing operation includes:

[0100] Identify the hot vertices in the shared block, and identify the M vertices with the highest degree among the hot vertices as global vertices, and the graph structures corresponding to the remaining hot vertices as core subgraphs; hot vertices are vertices whose degrees exceed the preset threshold, M is a positive integer; the set of global vertices is denoted as V global , the set of other hot vertices is denoted as V subgraph ;

[0101] Through the point-to-all query algorithm, the query results from each global vertex to each vertex in the shared block are obtained respectively, and the query results are used to build the query result table

[0102] Use the point-to-all query algorithm to obtain the query results from each vertex in the shared block to each global node, and use the query results to build a query result table

[0103] and In the point-to-point query with the same source and destination vertices, the query result is infinite. In practical applications, it can be set to a value greater than the maximum path length in the graph;

[0104] Initialize query result table T subgraph , which is used to record the query results between each vertex pair in the core subgraph; in the core subgraph, the path between two hot vertex pairs is called a hot path segment.

[0105] and T subgraph In , the row and column indices correspond to the source and destination vertices of the query. To keep the operation lightweight, unlike the query result table corresponding to the global vertex, T subgraph In the preprocessing stage, only null values ​​are created and initialized without direct calculation. This embodiment further finds that if a convergent query path is obtained between two vertices, any path segment extracted from the path also meets the convergence condition. Therefore, Tsubgraph The query results in will be gradually extracted and filled into the table during subsequent queries.

[0106] by Figure 5 As an example, (a) shows the specific graph structure. Based on the degree, vertex v6 ​​is identified as a global vertex, and vertices v2, v3, v5, v7, and v 10 Assign to the core subgraph; for global vertices, a query result table can be constructed and like Figure 5 As shown in (b); for the core subgraph, initialize the query result table T subgraph ,like Figure 5 As shown in (c) in .

[0107] Since the global vertex has the largest degree, each query path will most likely pass through the global vertex. The query result table established for the global vertex in this embodiment will be used to determine the initial boundary value of the point-to-point query, so that pruning can be effectively implemented at the initial moment of the query to avoid unnecessary calculations, thereby realizing the first layer of calculation sharing. Specifically, for any vertex v s is the source vertex, with vertex v d Point-to-point query Q(v s ,v d ), the initial boundary value bound is determined by:

[0108] From the query result table Get the vertex v s The query result R(v s ,v i ), that is, get the query result table Midpoint v s The query result of the corresponding row; v i ∈V global , V global Represents the global vertex set;

[0109] From the query result table Get the global vertex as the source vertex and the vertex v d The query result R(v i ,v d ), that is, get the query result table The vertex v in d The query results of the corresponding columns;

[0110] For the global vertex set V global For each global vertex v in i , calculate R(v s,v i )+R(v i ,v d ), and initialize the minimum calculation result as the point-to-point query Q(v s ,v d )’s boundary value is bound.

[0111] It is easy to understand that the boundary value bound will also be continuously updated in subsequent iterations.

[0112] As mentioned above, different queries often recalculate hot path segments. This embodiment uses the query result table T corresponding to the core subgraph. subgraph The query results of these hot path segments are recorded and shared for query, thus realizing the second layer of computing sharing. When traversing the path, if the active vertex traversed is a hot vertex v h , then we can get T subgraph The query result of finding the corresponding row in the table records the hot vertex v h To the query results of other hot vertices, if a query result is not empty, it means that a query task has traversed the corresponding hot path segment before, and obtained and in T subgraph The relevant query results are stored in , and the hot vertex corresponding to the non-empty query result is regarded as the hot vertex v h The outgoing neighbor vertices are directly used as hot path segments as edges, and the query results of the corresponding hot path segments are regarded as edge weights, which participate in the calculation and update of the active vertex set together with the regular outgoing neighbor vertices, thereby reducing the redundant calculation of the hot path segments and speeding up the traversal.

[0113] With the help of T subgraph In this embodiment, and Determine the point-to-point query Q(v s ,v d ) is bounded, point-to-point query Q(v s ,v d ) includes:

[0114] Step S1: traverse vertex v s The outgoing neighbor vertex v j , and calculate the vertex v s and its neighboring nodes v j The edge weight between the vertices v s to v j The query result R between them, and v j As the active vertex, get the active vertex set;

[0115] Step S2: If v s to v jIf it constitutes a valid path and R < bound, then update the value of the boundary value bound to the value of R; the valid path is a path with v s as the source vertex and v d as the destination vertex;

[0116] Step S3: If the active vertex set is empty, then go to step S7; otherwise, go to step S4;

[0117] Step S4: For each vertex v a in the active vertex set, if it is a hot vertex, then take each hot vertex as its out-neighbor vertex, and use the corresponding hot path segment query result in the query result table T subgraph as the edge weight; traverse the out-neighbor vertex v a of vertex v an , and calculate the edge weight W(v a between vertex v an and its each out-neighbor vertex v a ,v an ), and update the query result between vertex v a ,v an to v s according to R = R + W(v an );

[0118] Step S5: If R > bound, then prune the path between vertex v s to v an ; if there is a valid path in the remaining paths and the minimum query result corresponding to the valid path is less than the boundary value bound, then update the boundary value to the minimum query result corresponding to the valid path;

[0119] Step S6: Take the destination vertex of the remaining paths as the active vertex for the next iteration, obtain the active vertex set for the next iteration, and go to step S3;

[0120] Step S7: Determine the valid path with the minimum query result in the remaining paths as the query path of the point-to-point query Q(v s ,v d ), and determine the minimum query result as the final query result.

[0121] To effectively maintain the query result table T subgraph , in this embodiment, when performing a point-to-point query, if the finally obtained query path contains a hot path segment and the query result of this hot path segment is not recorded in T subgraph , then extract the query result of this hot path segment from the corresponding query path and record it in T subgraph .

[0122] Since any path segment of a convergent path itself remains convergent, taking PPSP as an example, if the shortest path between two vertices is obtained by query, any path segment on the path is the shortest path between the corresponding two vertices. In practical applications, if the known optimal query path contains multiple hot vertices, it means that there is at least one convergent path segment consisting of hot vertices at both ends. Optionally, in this embodiment, the path segment information can be extracted in the following manner:

[0123] When the query path passes through vertex v i When recording the path v s →v i The query results and the path v i The parent node of v is retained until a new better path reactivates v i ; In this way, when the final query converges, the query results and parent nodes of each relevant vertex can be obtained;

[0124] Based on the recorded results, from the destination vertex v d Start by following the converging path from the source vertex v s to v d and its corresponding parent vertex to extract the query result; then, repeat the previous step using the obtained parent vertex to obtain a vertex closer to the source vertex v s The vertices and related query results are retrieved until the query results of each path segment are obtained. After the query results of all path segments are extracted, the query results will be retrieved in T subgraph Update the query results at the corresponding position in .

[0125] It should be noted that due to the effective pruning operation used in point-to-point queries, a large number of invalid calculations are effectively avoided, and the proportion of activated vertices is very small. Therefore, the extraction process of hot path query results can be completed with extremely low time and space overhead.

[0126] In general, this embodiment proposes a data-driven graph query method to address the situation where traversal paths overlap in different point-to-point queries, which promotes shared access to the same graph structure for different queries, avoids data redundancy, and amortizes the cost of data access. On this basis, in order to address the problem of hot path segments being recalculated by a large number of queries, a two-layer computing sharing mechanism is proposed to accelerate query convergence by sharing the calculated values ​​from frequently accessed paths between different queries, further improving the execution efficiency of concurrent point-to-point queries. Experiments show that the graph computing method provided in this embodiment has an average performance that is 3.2 times higher than the most advanced point-to-point query system SGraph.

[0127] The following are some examples of practical applications of the data-driven graph query method provided by the embodiments of the present invention:

[0128] Application example 1: Optimizing logistics routes on a map:

[0129] Construct the graph structure data corresponding to the map, where the graph vertices represent key locations in the logistics network, such as warehouses, distribution centers, customer addresses, etc., and each vertex has a unique identifier; the edges represent the logistics connections between vertices, which can be roads, railways, etc.; the edge weights represent the various costs and considerations when transporting goods, such as distance, time, freight, etc. During the peak period of logistics, a large number of logistics demands are generated in a short period of time. The storage warehouses of these logistics goods and the destinations of users correspond to the source vertices and destination vertices of point-to-point queries respectively. These demands together constitute the query pool, and the graph data is the previously constructed map, which is stored in CSR and other ways. The path result is obtained through the above graph query method. If the map information does not change thereafter, the logistics route can continue to follow the path result.

[0130] Application Example 2: Recommending friends through social network analysis on a social application:

[0131] Graph structure data corresponding to a social network is constructed, where graph vertices represent individual users in the social network; edges represent relationships between users, such as friend relationships, follow relationships, etc.; edge weights represent the strength or quality of social relationships, which can be based on a variety of factors such as the number of common friends, interaction frequency, etc. In practical applications, by relying on the constructed graph structure data, other users who have common friends with the target user are found by analyzing the user's social network and recommending them as possible friends. The social platform can use the graph query method provided by the present invention to quickly recommend a large number of friends, so that new users can quickly establish their own circle of friends.

[0132] Application Example 3: Analyzing risk propagation in financial assessment on a payment application:

[0133] Construct graph structure data corresponding to payment applications, where graph vertices represent participants in the payment network, such as users, merchants, financial institutions, etc.; edges represent financial transactions or relationships, such as a payment, transfer, etc.; edge weights represent attributes such as transaction amount, transaction frequency, and transaction speed. The application in this scenario is mainly to monitor and analyze the flow of funds and risk propagation in the payment network in real time. The path obtained in the graph structure data for point-to-point queries is the transaction path. The path with the highest transaction frequency or the largest transaction amount can be tracked for certain merchants to monitor whether there is a possibility of financial risk. Some nodes may generate multiple transactions at the same time, which is suitable for the method of the present invention to track synchronously, quickly and accurately.

[0134] Embodiment 2:

[0135] A data-driven graph query device, comprising: a computer-readable storage medium and a processor;

[0136] The computer readable storage medium is used to store the computer program;

[0137] The processor is used to read the computer program stored in the computer-readable storage medium and execute the data-driven graph query method provided in the above-mentioned embodiment 1.

[0138] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data-driven graph query method, characterized in that: include: Dividing a query pool consisting of point-to-point queries to be executed into a plurality of query sets; The similarity of point-to-point queries within the same query set is higher than the similarity of point-to-point queries between different query sets, and the query set size does not exceed the preset number of concurrent queries N; Divide the graph structure to be queried into multiple shared blocks; Execute each query set in sequence; for the currently executed query set, execute the point-to-point queries in parallel, and during the execution process, calculate the priority of each shared tile in real time, and load the shared tiles into the last-level cache in order from high to low priority; Among them, at any moment, the priority of a shared tile is positively correlated with the number of peer-to-peer queries associated with it; The shared tile associated with a point-to-point query contains the active vertices required by the point-to-point query.

2. The data-driven graph query method according to claim 1, characterized in that: For any two point-to-point queries q i and q j , the similarity is: Among them, S ij Represents a point-to-point query q i and q j similarities between and They represent point-to-point queries q i The source and destination vertices of and They represent point-to-point queries q j The source and destination vertices of express and The distance between express and The distance between.

3. The data-driven graph query method according to claim 1, characterized in that: At any one time, the priority of shared tiles is: Where P represents the priority of the shared tile; N query Indicates the number of peer-to-peer queries associated with the shared tile; It represents the proportion of hot vertices in the active vertices contained in the shared tile. Hot vertices are vertices whose degrees exceed the preset threshold. ω1 and ω2 are preset weight coefficients.

4. The data-driven graph query method according to any one of claims 1 to 3, characterized in that: The edges of the graph structure to be queried are stored in an edge list in ascending order of source vertex IDs; In addition, the graph structure to be queried is divided into multiple shared blocks, including: According to the preset shared block size, the number of edges C that each shared block can accommodate is calculated, and the edges in the edge list are divided into a plurality of logical partitions in order; each logical partition contains C edges; Store the number of edges in each logical partition, the source vertex ID of the first edge, and the source vertex ID of the last edge into the corresponding data structure T block middle; The graph structure corresponding to each logical partition is a shared block.

5. The data-driven graph query method according to claim 4, characterized in that: The size of the shared block satisfies that: after removing the reserved redundant space of the last-level cache, the remaining space of the last-level cache is sufficient to store the shared block and its corresponding logical partition information.

6. The data-driven graph query method according to claim 5, characterized in that: The shared tile size meets the following requirements: Among them, S block Indicates the size of the shared tile, S graph represents the size of the logical partition, |V| represents the total number of vertices, and V vertex represents the storage cost of a single vertex, S LLC Indicates the size of the last-level cache, S reserve Indicates that redundant space is reserved.

7. The data-driven graph query method according to claim 6, characterized in that: Also includes: When the shared tiles are loaded into the last-level cache, preprocessing operations are performed; The pre-processing operation includes: Identify the hot vertices in the shared tiles, and identify the M vertices with the highest degrees among the hot vertices as global vertices, and the graph structures corresponding to the remaining hot vertices as core subgraphs; hot vertices are vertices whose degrees exceed a preset threshold, and M is a positive integer; Get the query results from each global vertex to each vertex in the shared block, and use the query results to build a query result table Get the query results from each vertex in the shared block to each global node respectively, and use the query results to build the query result table and In the point-to-point query, the query result is infinite if the source vertex and the destination vertex are the same; And, for any vertex v s is the source vertex, with vertex v d Point-to-point query Q(v s ,v d ), the initial boundary value bound is determined by: From the query result table Get the vertex v s The query result R(v s ,v i );v i ∈V global , V global Represents the global vertex set; From the query result table Get the global vertex as the source vertex and the vertex v d The query result R(v i ,v d ); For the global vertex set V global For each global vertex v in i , calculate R(v s ,v i )+R(v i ,v d ), and initialize the minimum calculation result as the point-to-point query Q(v s ,v d )’s boundary value is bound.

8. The data-driven graph query method according to claim 7, characterized in that: The pre-processing operation also includes: Initialize query result table T subgraph , used to record the query results between each vertex pair in the core subgraph; in the core subgraph, the path between two hot vertex pairs is called a hot path segment; And, the point-to-point query Q(v s ,v d ) includes: Step S1: traverse vertex v s The outgoing neighbor vertex v j , and calculate the vertex v s and its neighboring nodes v j The edge weight between the vertices v s to v j The query result R between them, and v j As the active vertex, get the active vertex set; Step S2: If the path from v s to v j forms a valid path and R < bound, then update the value of the boundary value bound to the value of R; The valid path is a path with v s as the source vertex and v d as the destination vertex; Step S3: If the active vertex set is empty, go to step S7; otherwise, go to step S4; Step S4: For each vertex v in the active vertex set a If it is a hot vertex, each hot vertex is used as its outgoing neighbor vertex, and the query result table T subgraph The corresponding hot path segment query result is used as the edge weight; traverse the vertex v a The outgoing neighbor vertex v an , and calculate the vertex v a and its neighbor vertices v an The edge weight W(v a ,v an ), and according to R=R+W(v a ,v an ) Update vertex v s to v an The query results between Step S5: If R>bound, then vertex v s to v an The paths between them are pruned; if there is a valid path among the remaining paths, and the minimum query result corresponding to the valid path is less than the boundary value bound, the boundary value is updated to the minimum query result corresponding to the valid path; Step S6: taking the destination vertex of the remaining path as the active vertex of the next iteration, obtaining the active vertex set of the next iteration, and proceeding to step S3; Step S7: Determine the valid path with the minimum query result among the remaining paths as the point-to-point query Q(v s ,v d ) and determine the minimum query result as the final query result; If the point-to-point query Q(v s ,v d ) includes a hot path segment, and the query result table T subgraph If the query result of the hot path segment is not recorded in the query result, then the point-to-point query Q(v s ,v d ) query path to extract the query result of the hot path segment and record it in the query result table T subgraph middle.

9. A data-driven graph query device, characterized in that: include: A computer readable storage medium and a processor; The computer readable storage medium is used to store a computer program; The processor is used to read the computer program stored in the computer-readable storage medium and execute the data-driven graph query method described in any one of claims 1 to 8.

Images