Cache method and system for storing knowledge graph relation table based on hadoop

Abstract

This invention relates to a caching method and system for storing relational tables in a Hadoop-based knowledge graph. Addressing the problem in existing technologies where join results or intermediate join results related to hot queries are stored as materialized views in additional tables, leading to slow response times for multi-table joins, this application proposes a caching framework. Leveraging Hadoop's efficient parallel processing capabilities, join results or intermediate results are cached in the file system, reducing the processing and maintenance burden on the database and thus accelerating the response time of multi-table joins. Specifically, this application caches query execution results or intermediate results of attribute tables as file blocks in the HDFS file system. This avoids directly storing results as materialized views in the database, reducing the database storage burden and utilizing Hadoop's parallel processing capabilities to reduce the join burden on attribute tables, thereby accelerating the response time of multi-table joins.

Description

Technical Field

[0001] This invention relates to the field of database technology, specifically to a caching method and system for storing knowledge graph relational tables based on Hadoop. Background Technology

[0002] Currently, among knowledge graph storage solutions, relational storage is widely used in academic research due to its earlier development, maturity, and good stability in the big data field. There are six main relational storage solutions for knowledge graphs: triple tables, horizontal tables, attribute tables, vertical partitioning, six-fold indexes, and DB2RDF. Among these, attribute tables are a further optimization of horizontal tables. They inherit the advantages of horizontal tables, overcome the self-join problem of triples, and solve the problem of excessive null values ​​in horizontal tables. Furthermore, intuitively, attribute tables better represent the directed graph structure of knowledge graphs using a relational schema, and are widely used in production systems in academia and industry. However, attribute tables still have some problems. For example, when queries involve joins with more than four tables, there is a significant join overhead, which greatly reduces the query response performance of attribute tables.

[0003] A common solution to this problem is to store the join results or intermediate join results involved in hot queries in an additional table as materialized views. However, this has the disadvantage of increasing the maintenance burden on relational databases and thus reducing the response speed of multi-table joins. Summary of the Invention

[0004] The purpose of this invention is to address the problem in existing technologies where the connection results or intermediate connection results involved in hot queries are stored in additional tables as materialized views, resulting in slow response times for multi-table joins. This invention proposes a caching method and system for storing knowledge graph relational tables based on Hadoop.

[0005] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0006] A caching method for storing knowledge graph relation tables based on Hadoop includes the following steps:

[0007] Step 1: The database receives user query requests and processes them using syntax analysis and lexical analysis to generate an Abstract Syntax Tree (AST).

[0008] Step 2: The database performs query optimization on the Abstract Syntax Tree (AST) and generates a query plan;

[0009] Step 3: The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments. At this time, the database is in a synchronization waiting state.

[0010] Then the query plan segments and the remaining query plans are sent to the Hadoop cache pool;

[0011] Step 4: Check if the cache is hit in the Hadoop cache pool. If the cache is not hit, Hadoop returns a cache miss message to the database.

[0012] If the cache is hit, the database receives the intermediate result returned by the Hadoop cache or after processing, and then continues to execute the remaining query plan until the final result is output;

[0013] Both cache hits and cache misses will interrupt the database synchronization wait;

[0014] Step 5: Return the final result to the user.

[0015] Furthermore, in step two, the database calls the query optimization module to optimize the abstract syntax tree (AST) and generate a query plan.

[0016] Furthermore, the specific steps of step three are as follows:

[0017] The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments.

[0018] Then, it is determined whether the query plan segment exists in the output result set of the materialized recommender module.

[0019] If it exists, select all the subtrees of the plan tree that are closest to the root node of the plan tree in the output result set as cached query plan segments; otherwise, do not cache and generate an empty query plan segment.

[0020] For cached query plan segments, output the remaining query plans based on this query plan segment;

[0021] Send the cached query plan segment or empty query plan segment, along with the remaining query plan based on this query plan segment, to the Hadoop cache pool;

[0022] The database is currently in a synchronization wait state.

[0023] Furthermore, the specific steps of step four are as follows:

[0024] In the Hadoop cache pool, it checks if there is a cache hit. If there is a cache hit, it determines whether the remaining query plan requires data from other tables. If not, the remaining query plan continues to execute in Hadoop. Then, the final result obtained from the Hadoop execution and the execution progress of the remaining query plan are returned to the database. If data is required, the execution progress of the remaining query plan is returned to the database.

[0025] If a cache miss occurs, Hadoop returns a cache miss message to the database.

[0026] Furthermore, if a cache miss occurs in step four, the following steps are executed:

[0027] Hadoop returns cache miss messages and miss plan segment messages to the database. The database then continues to execute the query plan obtained in step two, while sending the intermediate results corresponding to the miss plan segment to Hadoop and caching them.

[0028] Furthermore, the remaining query plan is executed in parallel in Hadoop.

[0029] A caching system based on Hadoop knowledge graph relation table storage, the system including a query plan generation module, a query plan segment generation module and a Hadoop caching module;

[0030] The query plan generation module is used to process user query requests received from the database using syntax analysis and lexical analysis, generate an abstract syntax tree (AST), optimize the AST for querying, and generate a query plan.

[0031] The query plan segment generation module is used to use the database to call the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments. At this time, the database is in a synchronous waiting state, and then the query plan segments and the remaining query plan are sent to the Hadoop cache pool.

[0032] The Hadoop caching module is used to determine whether a cache hit has occurred in the Hadoop cache pool. If a cache miss occurs, Hadoop returns a cache miss message to the database.

[0033] If the cache is hit, the database receives the Hadoop cache return or the intermediate result after processing, and then continues to execute the remaining query plan until the final result is output.

[0034] Both cache hits and cache misses will interrupt the database synchronization wait.

[0035] Finally, the final result is returned to the user.

[0036] Furthermore, in the query plan generation module, the database calls the query optimization module to optimize the abstract syntax tree (AST) and generate a query plan.

[0037] Furthermore, the query plan segment generation module specifically performs the following steps:

[0038] The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments.

[0039] Then, it is determined whether the query plan segment exists in the output result set of the materialized recommender module.

[0040] If it exists, select all the subtrees of the plan tree that are closest to the root node of the plan tree in the output result set as cached query plan segments; otherwise, do not cache and generate an empty query plan segment.

[0041] For cached query plan segments, output the remaining query plans based on this query plan segment;

[0042] Send the cached query plan segment or empty query plan segment, along with the remaining query plan based on this query plan segment, to the Hadoop cache pool;

[0043] The database is currently in a synchronization wait state.

[0044] Furthermore, the Hadoop caching module specifically performs the following steps:

[0045] In the Hadoop cache pool, it checks if there is a cache hit. If there is a cache hit, it determines whether the remaining query plan requires data from other tables. If not, the remaining query plan continues to execute in Hadoop. Then, the final result obtained from the Hadoop execution and the execution progress of the remaining query plan are returned to the database. If data is required, the execution progress of the remaining query plan is returned to the database.

[0046] If a cache miss occurs, perform the following steps:

[0047] Hadoop returns cache miss messages and miss plan segment tree messages to the database. The database then continues to execute the query plan obtained in step two, while sending the intermediate results corresponding to the miss plan segment trees to Hadoop and caching them.

[0048] The remaining query plan is executed in parallel in Hadoop.

[0049] The beneficial effects of this invention are:

[0050] This application proposes a caching framework that leverages Hadoop's efficient parallel processing capabilities to cache join results or intermediate results in the file system, reducing the processing and maintenance pressure on the database and thus accelerating the response speed of multi-table joins.

[0051] This application caches the query execution results or intermediate results of the attribute table in the HDFS file system as file blocks. This avoids storing the results directly in the database as materialized views, reducing the database storage burden. Furthermore, it utilizes Hadoop's parallel processing capabilities to reduce the connection burden of the attribute table, thereby accelerating the response speed of multi-table joins. Attached Figure Description

[0052] Figure 1 This is the overall flowchart of this application. Detailed Implementation

[0053] It should be noted that, where there is no conflict, the various embodiments disclosed in this application can be combined with each other.

[0054] Specific implementation method one: Refer to Figure 1 This embodiment describes a caching method for storing knowledge graph relation tables based on Hadoop, which includes the following steps:

[0055] Step 1: The database receives user query requests and processes them using syntax analysis and lexical analysis to generate an Abstract Syntax Tree (AST).

[0056] Step 2: The database performs query optimization on the Abstract Syntax Tree (AST) and generates a query plan;

[0057] Step 3: The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments. At this time, the database is in a synchronization waiting state.

[0058] Then the query plan segments and the remaining query plans are sent to the Hadoop cache pool;

[0059] Step 4: Check if the cache is hit in the Hadoop cache pool. If the cache is not hit, Hadoop returns a cache miss message to the database.

[0060] If the cache is hit, the database receives the intermediate result returned by the Hadoop cache or after processing, and then continues to execute the remaining query plan until the final result is output;

[0061] Both cache hits and cache misses will interrupt the database synchronization wait;

[0062] Step 5: Return the final result to the user.

[0063] Specific Implementation Method Two: This implementation method is a further explanation of Specific Implementation Method One. The difference between this implementation method and Specific Implementation Method One is that in step two, the database calls the query optimization module to optimize the abstract syntax tree (AST) and generate a query plan.

[0064] Specific Implementation Method Three: This implementation method is a further explanation of Specific Implementation Method Two. The difference between this implementation method and Specific Implementation Method Two is that the specific steps of step three are as follows:

[0065] The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments.

[0066] Then, it is determined whether the query plan segment exists in the output result set of the materialized recommender module.

[0067] If it exists, select all the subtrees of the plan tree that are closest to the root node of the plan tree in the output result set as cached query plan segments; otherwise, do not cache and generate an empty query plan segment.

[0068] For cached query plan segments, output the remaining query plans based on this query plan segment;

[0069] Send the cached query plan segment or empty query plan segment, along with the remaining query plan based on this query plan segment, to the Hadoop cache pool;

[0070] The database is currently in a synchronization wait state.

[0071] Specific Implementation Method Four: This implementation method is a further explanation of Specific Implementation Method Three. The difference between this implementation method and Specific Implementation Method Three is that the specific steps in step four are as follows:

[0072] In the Hadoop cache pool, it checks if there is a cache hit. If there is a cache hit, it determines whether the remaining query plan requires data from other tables. If not, the remaining query plan continues to execute in Hadoop. Then, the final result obtained from the Hadoop execution and the execution progress of the remaining query plan are returned to the database. If data is required, the execution progress of the remaining query plan is returned to the database.

[0073] If a cache miss occurs, Hadoop returns a cache miss message to the database.

[0074] Specific Implementation Method Five: This implementation method is a further explanation of Specific Implementation Method Four. The difference between this implementation method and Specific Implementation Method Four is that if a cache miss occurs in step four, the following steps are performed:

[0075] Hadoop returns cache miss messages and miss plan segment messages to the database. The database then continues to execute the query plan obtained in step two, while sending the intermediate results corresponding to the miss plan segment to Hadoop and caching them.

[0076] Specific Implementation Method Six: This implementation method is a further explanation of Specific Implementation Method Four. The difference between this implementation method and Specific Implementation Method Four is that the remaining query plan is executed in Hadoop in a parallel manner.

[0077] Specific implementation method seven: a caching system based on Hadoop knowledge graph relation table storage, the system including a query plan generation module, a query plan segment generation module and a Hadoop caching module;

[0078] The query plan generation module is used to process user query requests received from the database using syntax analysis and lexical analysis, generate an abstract syntax tree (AST), optimize the AST for querying, and generate a query plan.

[0079] The query plan segment generation module is used to use the database to call the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments. At this time, the database is in a synchronous waiting state, and then the query plan segments and the remaining query plan are sent to the Hadoop cache pool.

[0080] The Hadoop caching module is used to determine whether a cache hit has occurred in the Hadoop cache pool. If a cache miss occurs, Hadoop returns a cache miss message to the database.

[0081] If the cache is hit, the database receives the Hadoop cache return or the intermediate result after processing, and then continues to execute the remaining query plan until the final result is output.

[0082] Both cache hits and cache misses will interrupt the database synchronization wait.

[0083] Finally, the final result is returned to the user.

[0084] Specific Implementation Method Eight: This implementation method is a further explanation of Specific Implementation Method Seven. The difference between this implementation method and Specific Implementation Method Seven is that in the query plan generation module, the database calls the query optimization module to optimize the abstract syntax tree (AST) and generate a query plan.

[0085] Specific Implementation Method Nine: This implementation method is a further explanation of Specific Implementation Method Eight. The difference between this implementation method and Specific Implementation Method Eight is that the query plan segment generation module specifically performs the following steps:

[0086] The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. The suggestions on whether to cache the intermediate results of all query plan segments are the query plan segments.

[0087] Then, it is determined whether the query plan segment exists in the output result set of the materialized recommender module.

[0088] If it exists, select all the subtrees of the plan tree that are closest to the root node of the plan tree in the output result set as cached query plan segments; otherwise, do not cache and generate an empty query plan segment.

[0089] For cached query plan segments, output the remaining query plans based on this query plan segment;

[0090] Send the cached query plan segment or empty query plan segment, along with the remaining query plan based on this query plan segment, to the Hadoop cache pool;

[0091] The database is currently in a synchronization wait state.

[0092] Specific Implementation Method Ten: This implementation method is a further explanation of Specific Implementation Method Nine. The difference between this implementation method and Specific Implementation Method Nine is that the Hadoop caching module specifically performs the following steps:

[0093] In the Hadoop cache pool, it checks if there is a cache hit. If there is a cache hit, it determines whether the remaining query plan requires data from other tables. If not, the remaining query plan continues to execute in Hadoop. Then, the final result obtained from the Hadoop execution and the execution progress of the remaining query plan are returned to the database. If data is required, the execution progress of the remaining query plan is returned to the database.

[0094] If a cache miss occurs, perform the following steps:

[0095] Hadoop returns cache miss messages and miss plan segment tree messages to the database. The database then continues to execute the query plan obtained in step two, while sending the intermediate results corresponding to the miss plan segment trees to Hadoop and caching them.

[0096] The remaining query plan is executed in parallel in Hadoop.

[0097] In this application,

[0098] The materialized recommender module refers to the pre-trained or pre-equipped scheme for outputting which intermediate query results need to be materialized for a given query workload (existing implementation schemes include, but are not limited to, Zheng L, Shen Z, Wang H. GSBRL: Efficient RDF graph storage based on reinforcement learning[J]. WorldWide Web, 2022, 25(2): 763-784).

[0099] A query plan segment is a subtree of a (tree-like) query plan, also called a plan segment subtree. Typically, query plan segments are partitioned by the query optimizer in the database. The portion of the query plan excluding the plan segment subtree is called the remaining query plan.

[0100] Example:

[0101] 1. When the database receives a user query request, it performs syntax analysis and lexical analysis on the query to generate an Abstract Syntax Tree (AST);

[0102] 2. The database calls the query optimization module to generate a query plan from the Abstract Syntax Tree (AST);

[0103] 3. The database parses the query plan, segments it, and, referring to the output of the materialized recommender module, obtains suggestions on whether to cache the intermediate results of the query plan segments, as well as the remaining query plan based on these intermediate results.

[0104] a) By determining whether a query plan segment exists in the output result set of the materialized recommender module, if it exists, select all subtrees of plan segments that are closest to the root node of the plan tree within that output result set as cached query plan segments; otherwise, do not cache them and generate an empty query plan segment. For the query plan segments that are determined to be cached, it is also necessary to output the remaining query plans based on this query plan segment;

[0105] b) After this, the database synchronization will wait.

[0106] 4. The plan segment tree and remaining plan obtained in step 3 are handed over to Hadoop to determine whether the plan segment tree has been cached.

[0107] a) Cache hit. Analyzing the remaining query plan, if the remaining query plan does not require data from other tables (i.e., the operators in the remaining query plan are computational operators that do not depend on database table data), then the remaining query plan is considered to continue execution in Hadoop. Furthermore, it can be executed in Hadoop using optimization methods such as parallelization, and the final result obtained from the Hadoop run can be returned to the database. Regardless of whether the remaining query plan is executed in Hadoop, there may still be remaining operators that have not been processed; therefore, the execution progress of the remaining query plan must also be returned.

[0108] b) Cache miss. Hadoop returns a cache miss message to the data.

[0109] 5. Step 4 is completed (both cache hits and misses will interrupt the database synchronization wait), and the database synchronization wait ends.

[0110] a) If the cache is hit, the database receives the intermediate result returned or processed by the Hadoop cache, along with the remaining execution plan, and continues execution in the database until the final result is output.

[0111] b) If a cache miss occurs, the database receives a miss message that includes which plan segment tree was not hit. The database then continues executing the original full query plan, while sending the intermediate results corresponding to the missed plan segment trees to Hadoop for caching.

[0112] Once step 5 is completed, regardless of whether it's 5a) a cache hit or 5b) a cache miss, the final output of the original query plan will be returned to the user.

[0113] It should be noted that the specific embodiments are merely explanations and illustrations of the technical solution of the present invention and should not be used to limit the scope of protection. Any modifications made in accordance with the claims and specification of the present invention that are only partial should still fall within the protection scope of the present invention.

Claims

1. A cache method for storing Hadoop-based knowledge graph relation tables, characterized in that Includes the following steps: Step 1: The database receives user query requests and processes them using syntax analysis and lexical analysis to generate an Abstract Syntax Tree (AST). Step 2: The database performs query optimization on the Abstract Syntax Tree (AST) and generates a query plan; Step 3: The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results. At this time, the database is in a synchronization waiting state. Then the query plan segments and the remaining query plans are sent to the Hadoop cache pool; The specific steps of step three are as follows: The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results; Then, it is determined whether the query plan segment exists in the output result set of the materialized recommender module. If it exists, all the plan segment subtrees that are closest to the root node of the plan tree in the output result set are selected as the cached query plan segments. Otherwise, do not cache and generate an empty query plan segment; For cached query plan segments, output the remaining query plans based on this query plan segment; Send the cached query plan segment or empty query plan segment, along with the remaining query plan based on this query plan segment, to the Hadoop cache pool; The database is currently waiting for synchronization. Step 4: Check if the cache is hit in the Hadoop cache pool. If the cache is not hit, Hadoop returns a cache miss message to the database. If the cache is hit, the database receives the intermediate result returned by the Hadoop cache or after processing, and then continues to execute the remaining query plan until the final result is output; Both cache hits and cache misses will interrupt the database synchronization wait; Step 5: Return the final result to the user. 2.The cache method of Hadoop-based knowledge graph relation table storage according to claim 1, characterized in that In step two, the database calls the query optimization module to optimize the abstract syntax tree (AST) and generate a query plan.

3. The caching method for storing knowledge graph relation tables based on Hadoop according to claim 2, characterized in that... The specific steps of step four are as follows: In the Hadoop cache pool, it checks if there is a cache hit. If there is a cache hit, it determines whether the remaining query plan requires data from other tables. If not, the remaining query plan continues to execute in Hadoop. Then, the final result obtained from the Hadoop execution and the execution progress of the remaining query plan are returned to the database. If data is required, the execution progress of the remaining query plan is returned to the database. If a cache miss occurs, Hadoop returns a cache miss message to the database.

4. The caching method for storing knowledge graph relation tables based on Hadoop according to claim 3, characterized in that... If a cache miss occurs in step four, the following steps are executed: Hadoop returns cache miss messages and miss plan segment messages to the database. The database then continues to execute the query plan obtained in step two, while sending the intermediate results corresponding to the miss plan segment to Hadoop and caching them.

5. The caching method for storing knowledge graph relation tables based on Hadoop according to claim 3, characterized in that... The remaining query plan is executed in parallel in Hadoop.

6. A caching system based on Hadoop knowledge graph relation table storage, characterized in that... The system includes a query plan generation module, a query plan segment generation module, and a Hadoop caching module; The query plan generation module is used to process user query requests received from the database using syntax analysis and lexical analysis, generate an abstract syntax tree (AST), optimize the AST for querying, and generate a query plan. The query plan segment generation module is used to use the database to call the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on the intermediate results. At this time, the database is in a synchronous waiting state, and then the query plan segments and the remaining query plan are sent to the Hadoop cache pool. The query plan segment generation module specifically performs the following steps: The database calls the materialized recommender module to parse and segment the query plan, obtain suggestions on whether to cache the intermediate results of all query plan segments, and the remaining query plan based on these intermediate results; Then, it is determined whether the query plan segment exists in the output result set of the materialized recommender module. If it exists, all the plan segment subtrees that are closest to the root node of the plan tree in the output result set are selected as the cached query plan segments. Otherwise, do not cache and generate an empty query plan segment; For cached query plan segments, output the remaining query plans based on this query plan segment; Send the cached query plan segment or empty query plan segment, along with the remaining query plan based on this query plan segment, to the Hadoop cache pool; The database is currently waiting for synchronization. The Hadoop caching module is used to determine whether a cache hit has occurred in the Hadoop cache pool. If a cache miss occurs, Hadoop returns a cache miss message to the database. If the cache is hit, the database receives the Hadoop cache return or the intermediate result after processing, and then continues to execute the remaining query plan until the final result is output. Both cache hits and cache misses will interrupt the database synchronization wait. Finally, the final result is returned to the user.

7. The caching system for storing knowledge graph relation tables based on Hadoop according to claim 6, characterized in that... In the query plan generation module, the database calls the query optimization module to optimize the abstract syntax tree (AST) and generate a query plan.

8. The caching system for storing knowledge graph relation tables based on Hadoop according to claim 7, characterized in that... The Hadoop caching module specifically performs the following steps: In the Hadoop cache pool, it checks if there is a cache hit. If there is a cache hit, it determines whether the remaining query plan requires data from other tables. If not, the remaining query plan continues to execute in Hadoop. Then, the final result obtained from the Hadoop execution and the execution progress of the remaining query plan are returned to the database. If data is required, the execution progress of the remaining query plan is returned to the database. If a cache miss occurs, perform the following steps: Hadoop returns cache miss messages and miss plan segment tree messages to the database. The database then continues to execute the query plan obtained in step two, while sending the intermediate results corresponding to the miss plan segment trees to Hadoop and caching them. The remaining query plan is executed in parallel in Hadoop.

Images