Graph database query method based on runtime filtering

Abstract

The invention discloses a graph database query method based on runtime filtering, which belongs to the technical field of graph data, and comprises the following steps: analyzing a graph database query request to obtain an original physical execution plan; identifying each connection operator meeting a runtime filter generation condition as a target connection operator, and verifying that a scanning operator capable of receiving the runtime filter exists at the upstream of the detection side of each target connection operator; a runtime filter transmission channel is established, and the output end of the construction side of each target connection operator is connected to the input end of the corresponding scanning operator; during operation, the construction side operator completes data processing and then sends the data to the runtime filter, and the corresponding scanning operator receives the runtime filter, executes optimized scanning by using the structured storage characteristics of the graph storage engine, and outputs graph elements matched with the runtime filter. According to the method, the RuntimeFilter is introduced into the physical execution plan of the graph database query request, so that the data scanning amount can be greatly reduced, and the query speed is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of graph data, and in particular, to a graph database query method based on runtime filtering. Background Art

[0002] A graph database is a database system that stores data in a graph structure. It models data and their interrelationships through nodes (vertices) and relationships (edges), and can naturally and efficiently represent complex associations between entities. It is widely used in fields such as social networks, recommendation systems, and knowledge graphs.

[0003] However, in the actual application process, with the continuous growth of the scale of graph data, the query performance of traditional graph databases faces significant challenges. Specifically, when performing complex graph traversal and pattern matching operations, the problem is particularly prominent. Graph traversal operations need to access nodes and edges in the graph according to a certain strategy to obtain a specific path or subgraph; pattern matching operations need to find substructures in the graph that match a given pattern. These operations usually require repeated scanning and access of a large number of nodes and edges, resulting in an extended response time and increased consumption of system resources, thereby affecting the overall query efficiency.

[0004] Therefore, how to improve the query performance in a large-scale graph data environment has become one of the key problems that need to be solved urgently in the current development of graph database technology. Summary of the Invention

[0005] The purpose of the present invention is to provide a graph database query method based on runtime filtering to solve the problem of low query efficiency of large-scale graph databases in the prior art.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A graph database query method based on runtime filtering of the present application includes the following steps:

[0008] Parse the graph database query request to obtain the original physical execution plan, where the original physical execution plan includes at least one join operator based on hash join for processing the adjacency join of edges and vertices;

[0009] Identify each join operator that meets the runtime filter generation condition as the target join operator, and verify that there is a scan operator that can receive the runtime filter upstream of the probe side of each target join operator;

[0010] Establish a runtime filter transmission channel, and connect the output end of the build side of each target join operator to the input end of the corresponding scan operator;

[0011] At runtime, the build-side operator sends the runtime filter after completing data processing. The corresponding scanning operator receives the runtime filter and uses the structured storage characteristics of the graph storage engine to perform optimized scanning and output graph elements that match the runtime filter.

[0012] Preferably, the runtime filter generation condition includes:

[0013] The connection condition of the connection operator is the equivalue connection between the edge's starting point identifier and the point identifier or the equivalue connection between the point identifier and the edge's end point identifier.

[0014] Preferably, the runtime filter is a primary key set of a hash table generated in a hash join construction phase.

[0015] Preferably, when the scanning operator is an edge scanning operator, the received runtime filter is a set of edge start point identifiers;

[0016] When the scanning operator is a point scanning operator, the received runtime filter is a set of edge endpoint identifiers.

[0017] Preferably, the method further comprises:

[0018] The scanning operator enters a blocked waiting state before receiving the runtime filter, and is unblocked and starts the optimization scan after receiving the runtime filter.

[0019] Preferably, the optimized scan includes prefix scanning based on ordered storage of an adjacency list, range scanning based on a point identifier cluster index, or variable table filtering based on a Bloom filter.

[0020] Preferably, the runtime filter transmission channel is an asynchronous transmission channel across different pipelines in the original physical execution plan, and is implemented through shared memory.

[0021] An electronic device comprises a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a graph database query method based on runtime filtering as described in any one of the above.

[0022] A computer-readable storage medium storing a computer program, wherein the computer program enables a computer to implement a graph database query method based on runtime filtering as described in any one of the above when executed.

[0023] A computer program product includes a computer program or instructions, which, when executed by a processor, implements a graph database query method based on runtime filtering as described in any one of the above.

[0024] This application has the following beneficial effects:

[0025] By introducing RuntimeFilter into the physical execution plan of the graph database query request in this application, not only can the amount of data scanned be greatly reduced, thereby improving the query speed. Through RuntimeFilter, the graph database can also significantly improve the query performance under massive data, facilitating scenarios such as social network analysis, implementation recommendation, and financial risk control. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0027] Figure 1 is a flowchart of a graph database query method based on runtime filtering provided by an embodiment of the present application;

[0028] Figure 2 is a schematic diagram of the logical execution plan of the query request provided by an embodiment of the present application;

[0029] Figure 3 is a schematic diagram after introducing RuntimeFilter into the original physical execution plan of the query request provided by an embodiment of the present application;

[0030] Figure 4 is a schematic diagram of the original physical execution plan of the query request provided by an embodiment of the present application;

[0031] Figure 5 is a schematic diagram of an electronic device for implementing a graph database query method based on runtime filtering provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] To make the technical solutions of the present application clearer, the following will further describe the present invention in detail with reference to the drawings and specific embodiments. The terms "first", "second", etc. in the claims and the description of the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such terms can be interchanged under appropriate circumstances. This is only a way of distinguishing objects with the same attributes when describing the embodiments of the present application. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that a process, method, system, product, or device including a series of units does not have to be limited to those units, but may include other units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0033] Glossary:

[0034] Logical Execution Plan: In a graph database, a logical execution plan is a high-level description of the logical operation steps expressed by a query statement. It does not concern itself with specific implementation details or the underlying data storage method, but rather focuses on conceptually meeting the query requirements.

[0035] Physical Execution Plan: In a graph database, the physical execution plan is the specific implementation of the logical execution plan. It details how to efficiently execute the logical plan, taking into account factors such as the actual architecture of the database system, index usage, data distribution, and hardware characteristics.

[0036] Figure 1 It is a flowchart showing a graph database query method based on runtime filtering provided by this embodiment. As Figure 1 shown, a graph database query method based on runtime filtering can be executed by a computer or a server, or software on a computer or a server, and specifically includes the following steps:

[0037] S110. Parse the graph database query request to obtain the original physical execution plan, which contains at least one join operator based on hash join for processing the adjacency join of edges and vertices;

[0038] S120. Identify each join operator that meets the runtime filter generation conditions as the target join operator, and verify that there is a scan operator that can receive the runtime filter upstream of the probe side of each target join operator;

[0039] S130. Establish a runtime filter transmission channel, and connect the output end of the build side of each target join operator to the input end of the corresponding scan operator;

[0040] S140. At runtime, after the build side operator completes data processing, it sends the runtime filter, and the corresponding scan operator receives the runtime filter and performs an optimized scan using the structured storage characteristics of the graph storage engine, and outputs graph elements that match the runtime filter.

[0041] In this embodiment, after obtaining a graph database query request, first parse this query request to obtain its corresponding logical execution plan, and then convert this logical execution plan into an original physical execution plan.

[0042] Suppose the given graph database query request is MATCH(p1:Person)-[e1:KNOWS]->(p2:Person), and the purpose of this query request is to query all vertices and edges in the graph database that meet the following conditions:

[0043] 1. Two points p1 and p2, both with the label Person.

[0044] 2. There is an edge of type KNOWS between p1 and p2, and this relationship points from p1 to p2.

[0045] That is, MATCH(p1:Person)-[e1:KNOWS]->(p2:Person) means finding all patterns in the graph database where p1 knows p2.

[0046] Figure 2 It is a schematic diagram of the logical execution plan of the query request provided in this embodiment, where the arrow represents the data flow direction. As Figure 2 shown, after obtaining this query request, the graph database will first parse it into a logical execution plan, which specifically includes the following parts:

[0047] 1. p1 point scan operator NodeScan(p1): Its function is to find p1, that is, all points with the label Person.

[0048] 2. e1 edge scan operator EdgeScan(e1): Its function is to find e1, that is, all edges of type KNOWS.

[0049] 3. First join operator Join1: The condition is left_node_id(e1)=element_id(p1), that is, finding all data that satisfies the start identifier of the e1 edge is equal to the identifier of the p1 point.

[0050] 4. p2 point scan operator NodeScan(p2): Its function is to find p2, that is, all points with the label Person.

[0051] 5. Second join operator Join2: The condition is element_id(p2)=right_node_id(e1), that is, finding all data that satisfies the end identifier of the e1 edge is equal to the identifier of the p2 point.

[0052] It should be noted here that in a graph database (such as Neo4j), an edge is directional. It connects two nodes and has a clear start node and end node. In this embodiment, the left endpoint left_node of the edge is defined as the start point, and the right endpoint right_node of the edge is defined as the end point. In specific practice, the start and end points of the edge can also be adjusted according to actual needs without limitation.

[0053] Figure 3It is a schematic diagram of the original physical execution plan of the query request provided in this embodiment, where the green arrow represents HashBuild sending a hash table (HashTable) to HashProbe. As Figure 3 shown, for each operator in the logical execution plan, select an appropriate physical operator and convert it. For example, the first join operator Join1 is selected to be implemented using a hash join (HashJoin), which will be converted into two physical operators: the first build stage (HashBuild1) and the first probe stage (HashProbe1). Similarly, the second join operator Join2 will be converted into two physical operators: the second build stage (HashBuild2) and the second probe stage (HashProbe2). Then, corresponding to the original physical execution plan, organize each operator into different pipelines (Pipelines). For example, NodeScan(p1)->HashBuild1 is Pipeline2, which means that in Pipeline2, first scan all points marked as Person to obtain all p1 points, and then use the information of p1 points to construct a hash table HashTable1; EdgeScan(e1) ->HashProbe1->HashBuild is Pipeline1, which means that in Pipeline1, first scan all edges marked as KNOWS to obtain all e1 edges, and then use the information of e1 edges as the input for hash probing, that is, the information of e1 edges will be used to find out whether it matches the key values in the existing hash table HashTable1. Then, based on the results obtained from hash probing, construct a new hash table HashTable2; NodeScan(p2)->HashProbe is Pipeline0, which means that in Pipeline0, first scan all points marked as Person to obtain all p2 points, and then use the information of p2 points as the input for hash probing, that is, the information of p2 points is used to find out whether it matches the key values in the existing hash table HashTable2.

[0054] It should be noted here that in a graph database, a Pipeline is a mechanism for data processing and query optimization. By connecting multiple operations or query steps in series and executing them in an efficient, pipelined manner, it can be analogized to an assembly line in a factory. The output of each step is directly used as the input of the next step, avoiding the persistent storage of intermediate results.

[0055] In this embodiment, Join1 is split into two parts: HashBuild1 and HashProbe1, and Join2 is split into two parts: HashBuild2 and HashProbe2. Among them, HashBuild is used to generate a hash table HashTable based on the data received from the upstream (assuming there are m pieces of data) for probing during HashProbe; HashProbe is used to traverse all the data received from the upstream (assuming there are n pieces of data), check whether each record can match the records in the HashTable passed from HashBuild, and output the relevant data if there is a successful match, or skip it if not.

[0056] It should be noted here that HashProbe needs to wait until HashBuild is completed before it can proceed. Otherwise, the Join result will be incorrect due to incomplete records in the HashTable.

[0057] After obtaining the original physical execution plan, first determine whether each Join meets the generation conditions of the runtime filter (RuntimeFilter).

[0058] Furthermore, the generation conditions of the runtime filter include:

[0059] The join condition of the join operator is an equal join between the start identifier of the edge and the point identifier or an equal join between the point identifier and the end identifier of the edge.

[0060] Only when the Join condition is an equal join between the endpoint identifier of the edge and the identifier of the point can the RuntimeFilter be used. In this embodiment, only when the Join condition satisfies the equal join between the start identifier of the e1 edge and the p1 point identifier left_node_id(e1)=element_id(p1) or the equal join between the end identifier of the e1 edge and the p2 point identifier element_id(p2)=right_node_id(e1) can the RuntimeFilter be used, and the Join operator that meets the conditions is called the target Join operator.

[0061] In this embodiment, both Join1 and Join2 meet the generation conditions of the runtime filter and are both target Join operators.

[0062] Then, it is detected whether there is a scan operator that can receive the RuntimeFilter upstream of the probe side of each target Join operator. This scan operator can be an edge scan operator or a point scan operator. In this embodiment, the upstream scan operator of HashProbe1 of Join1 is EdgeScan(e1), and the upstream scan operator of HashProbe2 of Join2 is NodeScan(p2), both of which meet the conditions.

[0063] When all conditions are met, a RuntimeFilter transmission channel is established from the output end of the connected HashBuild to the input end of the upstream scan operator of HashProbe. Figure 4 It is a schematic diagram of the original physical execution plan of the query request provided in this embodiment after introducing the RuntimeFilter. The green arrow represents that HashBuild sends the HashTable to HashProbe, and the orange arrow represents that HashBuild sends the RuntimeFilter to the operator Scan upstream of the corresponding HashProbe. As Figure 4 shown, in this embodiment, a RuntimeFilter transmission channel is established that connects the output end of HashBuild1 to the input end of the upstream scan operator EdgeScan(e1) of HashProbe1, and a RuntimeFilter transmission channel is established that connects the output end of HashBuild2 to the input end of the upstream scan operator NodeScan(p2) of HashProbe2.

[0064] It can be understood that the RuntimeFilter transmission channels established in this embodiment are all asynchronous transmission channels across different pipelines in the original physical execution plan. As Figure 4 shown, one RuntimeFilter crosses Pipeline2 and Pipeline1, and one RuntimeFilter crosses Pipeline1 and Pipeline0. And the asynchronous transmission enables operations that generate the RuntimeFilter (such as scanning and aggregation) not to wait for the filter to be applied to subsequent operations, and vice versa, improving the decoupling between pipelines. At the same time, it is implemented through shared memory and can also reduce latency.

[0065] After the construction of the RuntimeFilter transmission channel is completed, the query is executed according to the physical execution plan with the RuntimeFilter introduced. In this embodiment, the generated RuntimeFilter is different according to the different upstream scan operators of HashProbe.

[0066] Further, when the scan operator is an edge scan operator, the received runtime filter is a set of start point identifiers of the edges;

[0067] When the scan operator is a point scan operator, the received runtime filter is a set of end point identifiers of the edges.

[0068] First, all p1 points with the label Person are scanned in the graph storage engine according to the NodeScan(p1) operator. Then, in Join1, HashBuild1 constructs a hash table HashTable1 with the identifiers of all p1 points as the primary key, and stores the identifiers of these points in a separate set f as the RuntimeFilter. When the data processing of HashBuild1 is completed, f is transmitted to the scan operator EdgeScan(e1) at the bottom layer of the pipeline where HashProbe1 is located, that is, upstream of HashProbe1. Since edges in the graph database are saved in the form of start point identifier -> end point identifier, if a given point is provided, all edges starting from the given point can be obtained through prefix scanning. Therefore, when EdgeScan(e1) receives f, it can perform optimized scanning based on each point identifier in f to obtain all edges e1 with the start point identifier of p1. In this embodiment, the optimized scanning is specifically a prefix scanning based on the ordered storage of the adjacency list. Prefix scanning is usually used to traverse a specific part of the graph, such as all edges or paths starting from a certain vertex. The ordered storage of the adjacency list makes this traversal more efficient because it can quickly locate the starting vertex and traverse along its neighbor linked list. Then, HashTable1 is transmitted to HashProbe1. HashProbe1 calculates the hash value of the left_node_id of each edge e1 and matches it with the hash value of the p1 point identifier in HashTable1. As long as there is a matching item in HashTable1, the (p1, e1) pair is output, so as to find all data that satisfies left_node_id(e1)=element_id(p1) and transmit it to the downstream. Here, the downstream is HashBuild2 of Join2.

[0069] In this embodiment, f is only a subset of all point sets. The amount of edge data obtained through prefix scanning is less than that obtained by directly performing a full table scan using EdgeScan(e1), which in turn reduces the amount of data processed by HashProbe1, thereby improving the overall query performance.

[0070] In Join2, it first receives all the data that satisfies left_node_id(e1)=element_id(p1) from the upstream as the input of HashProbe2, obtains the right_node_id of edge e1, and constructs HashTable2. And it stores the identifiers of these endpoints in a separate set g as a RuntimeFilter. When the HashBuild2 data processing is completed, it transmits g to the bottommost operator of the pipeline where HashProbe2 is located, namely the scan operator NodeScan(p2) upstream of HashProbe2. NodeScan(p2) performs a prefix scan based on each endpoint identifier in g to obtain all p2 points that are the same as the right_node_id of edge e1. Then, it transmits the data of p2 to HashProbe2. HashProbe2 matches the point identifier of each p2 with the right_node_id in HashTable2. As long as there is a matching item in HashTable1, it outputs the (p1, e1, p2) pair, thus obtaining all the data that satisfies element_id(p2)=right_node_id(e1).

[0071] In this embodiment, HashProbe2 only needs to process the filtered p2 points, reducing the amount of data to be processed. And the identifiers of all p2 points are necessarily the right_node_id of edge e1, with a 100% matching success rate, improved matching efficiency, and reduced memory and CPU overhead.

[0072] Furthermore, the optimized scan includes a prefix scan based on the ordered storage of the adjacency list, a range scan based on the clustered index of the point identifier, or a list filter based on the Bloom filter.

[0073] In this embodiment, in addition to the prefix scan described in the above embodiment, the optimized scan can also be a range scan based on the clustered index of the point identifier or a list filter based on the Bloom filter. Among them, the clustered index stores the index and data in one structure, making the data physically ordered. In graph data queries, if the point identifier (such as vertex ID) is the key of the clustered index, then range queries (such as querying all vertices with IDs within a certain range) will be very efficient; the Bloom filter is a probabilistic data structure with high space efficiency, used to determine whether an element belongs to a set. In graph data queries, the Bloom filter can be used to quickly determine whether a certain vertex or edge exists in the graph without actually accessing the graph data. That is to say, in this embodiment, by optimizing the data storage or introducing auxiliary data structures such as indexes and filters, the amount of data to be accessed during the query process can be reduced, improving the query efficiency. Therefore, the specific scan method is not limited here and can be set according to actual needs.

[0074] Furthermore, the scanning operator enters a blocked waiting state before receiving the runtime filter, and is unblocked and starts the optimization scan after receiving the runtime filter.

[0075] To ensure the accuracy of the Join results, the scanning operator will remain blocked before receiving the RuntimeFilter. Only after receiving the RuntimeFilter will it be unblocked and start the optimized scan.

[0076] This embodiment introduces RuntimeFilter into the physical execution plan of the graph database query request, which can not only greatly reduce the amount of data scanning, but also improve the query speed. Through RuntimeFilter, the graph database can also greatly improve the query performance under massive data, which is helpful for social network analysis, implementation recommendation, financial risk control and other scenarios.

[0077] like Figure 5 As shown, this embodiment also provides a computer device, including a memory, a processor, and a computer program stored in the memory, and the processor executes the computer program to implement the steps of the above-mentioned graph database query method based on runtime filtering.

[0078] The computer device can be a server or a terminal. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps of a graph database query method based on runtime filtering are implemented.

[0079] This embodiment also provides a computer-readable storage medium on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the graph database query method based on runtime filtering as described above are implemented.

[0080] This embodiment also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of a graph database query method based on runtime filtering as described above.

[0081] These computer-readable programs / instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing devices, thereby producing a machine such that when these instructions are executed by the processor of the computer or other programmable data processing devices, a device is produced that implements the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which causes the computer, programmable data processing device, and / or other devices to work in a specific manner. Thus, the computer-readable medium storing the instructions includes a manufactured article that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0082] The above-described embodiments merely represent several implementation manners of the present invention. The description thereof is relatively specific and detailed, but should not be construed as a limitation on the scope of the patent for the present invention. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be subject to the appended claims.

Claims

1. A graph database query method based on runtime filtering, characterized in that, The following steps are involved: Parsing the graph database query request to obtain an original physical execution plan, wherein the original physical execution plan includes at least one hash connection-based connection operator for processing adjacency connections between edges and points; Identify each connection operator that meets the runtime filter generation condition as a target connection operator, and verify that there is a scanning operator that can receive the runtime filter upstream of the detection side of each target connection operator; Establish a runtime filter transmission channel to connect the output of each target connection operator construction side to the input of the corresponding scan operator; At runtime, the build-side operator sends the runtime filter after completing data processing. The corresponding scanning operator receives the runtime filter and uses the structured storage characteristics of the graph storage engine to perform optimized scanning and output graph elements that match the runtime filter.

2. The graph database query method based on runtime filtering according to claim 1, wherein The runtime filter generation conditions include: The connection condition of the connection operator is the equivalue connection between the edge's starting point identifier and the point identifier or the equivalue connection between the point identifier and the edge's end point identifier.

3. The method for querying a graph database based on runtime filtering according to claim 2, wherein The runtime filter is a primary key set of the hash table generated in the hash connection construction phase.

4. A graph database query method based on runtime filtering according to claim 3, characterized in that: When the scanning operator is an edge scanning operator, the received runtime filter is a set of edge start point identifiers; When the scanning operator is a point scanning operator, the received runtime filter is a set of edge endpoint identifiers.

5. The graph database query method based on runtime filtering according to claim 4, wherein, The method further comprises: The scanning operator enters a blocked waiting state before receiving the runtime filter, and is unblocked and starts the optimization scan after receiving the runtime filter.

6. The graph database query method based on runtime filtering according to claim 5, characterized in that, The optimized scan includes prefix scan based on ordered storage of an adjacency list, range scan based on a point identifier cluster index, or variable list filtering based on a Bloom filter.

7. A graph database query method based on runtime filtering according to claim 1, characterized in that, The runtime filter transmission channel is an asynchronous transmission channel across different pipelines in the original physical execution plan, and is implemented through shared memory.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement a graph database query method based on runtime filtering as described in any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program enables a computer to implement a graph database query method based on runtime filtering as described in any one of claims 1 to 7 when executed.

10. A computer program product, comprising a computer program or instructions, which, when executed by a processor, implements a graph database query method based on runtime filtering as described in any one of claims 1 to 7.

Images