Query acceleration method, system and device for extracting data based on SQL query template

By generating parameter dimension tables based on SQL query templates and rewriting relational projection SQL, and using HBase to store query instances and results, the problem of insufficient performance in large-scale report queries is solved, achieving ultimate query performance and high query efficiency.

CN116467341BActive Publication Date: 2026-05-05ZHEJIANG DAYING TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
ZHEJIANG DAYING TECH CO LTD
Filing Date
2023-04-18
Publication Date
2026-05-05

AI Technical Summary

Technical Problem

Existing technologies cannot guarantee optimal query performance for large-scale data report queries. Pre-calculation solutions require professional maintenance and cannot adapt to changes in query dates. Materialized views are not suitable for all data sources.

Method used

By extracting data based on SQL query templates, a parameter dimension table is generated and the relational projection SQL is rewritten. The query instances and results are stored in the HBase database, enabling dynamic parameter scheduling and execution, and data can be obtained directly from the HBase query results or the original database.

Benefits of technology

It achieves optimal query performance in large-scale data report queries, reduces the need for professional maintenance, adapts to changes in query dates, and improves query efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116467341B_ABST
    Figure CN116467341B_ABST
Patent Text Reader

Abstract

This invention discloses a query acceleration method based on SQL query templates for extracting data, comprising the following steps: extracting an initial SQL query pattern for query instances based on historical behavioral data; generating a parameter dimension table based on the dynamic parameters and historical parameter values; rewriting the initial SQL query pattern and parameter dimension table according to a preset model to generate relational projection SQL; scheduling the relational projection SQL according to date parameters, executing the relational projection SQL to obtain execution results, converting the execution results into all query instances corresponding to the query pattern, and storing the query instances and their corresponding results in a one-to-one mapping in an HBase database; checking whether the result corresponding to the query instance exists in the HBase database, and if so, requesting the HBase database. This invention, by utilizing SQL query templates and historical parameter values ​​to materialize and rewrite SQL, and scheduling by date, can pre-calculate all SQL instances that may be queried, thereby achieving ultimate query performance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of big data query technology, and in particular to a query acceleration method, system and device for extracting data based on SQL query templates. Background Technology

[0002] In existing technologies, data query acceleration techniques are frequently used in reporting scenarios because report queries typically require extremely high response performance, such as opening within 3 seconds. After deducting the time spent on page rendering and other processes, the actual SQL query itself only takes about 1 second. Currently, there are three solutions for accelerating data queries: pre-computation; optimization at the storage and computation layer (such as columnar storage or MPP computing engines); or caching hot query results so they can be directly returned for subsequent queries.

[0003] For reports with large amounts of data, the second approach, while computationally efficient, cannot meet the highest performance requirements. The third approach can solve the performance issues of repeated queries, but it cannot handle scenarios where the query date changes. Under relatively fixed query patterns, pre-computation is a better solution. Currently, pre-computation is generally achieved through two methods: using cubes, such as Kylin, to pre-compute a specific aggregate column based on certain dimension columns; or building materialized views.

[0004] The current shortcomings of pre-computation are: it cannot guarantee optimal query performance for all reports; pre-computation configurations, such as Kylin cubes and materialized views, require support from specialized data personnel and frequent maintenance; even with such experts, optimal performance for all report queries cannot be guaranteed. For example, with Kylin cubes, if the configured dimension and aggregation columns differ from the actual query, secondary calculations will occur (in real-world scenarios, the personnel configuring reports and those building and configuring cubes for data model construction are separate departments), leading to performance failures. Building materialized views presents the same problem, and not all data sources support materialized views. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a query acceleration method, system, and apparatus for extracting data based on SQL query templates.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] A query acceleration method based on SQL query templates for extracting data includes the following steps:

[0008] The initial SQL query pattern for extracting query instances based on historical behavior data includes dynamic parameters and historical parameter values.

[0009] A parameter dimension table is generated based on the dynamic parameters and historical parameter values;

[0010] The initial SQL query pattern and the parameter dimension table are rewritten according to a preset model to generate relational projection SQL;

[0011] Based on dynamic parameters, the relational projection SQL is scheduled and executed to obtain the execution result. The execution result is converted into all query instances corresponding to the query mode, and the query instances and their corresponding results are mapped one-to-one and stored in the HBase database.

[0012] Check if the result corresponding to the instance to be queried exists in the HBase database. If it does, then request the HBase database.

[0013] As one possible implementation, the initial SQL query pattern for extracting query instances based on historical behavior data includes the following steps:

[0014] Obtain historical behavior data, which includes all data of the query instance;

[0015] Obtain all data of the query instance, wherein the all data includes the query instance, and infer dynamic parameters and historical combinations of dynamic parameters based on the query instance;

[0016] All data are clustered and extracted according to a preset clustering model to obtain the corresponding initial SQL query pattern. The preset clustering model is based on dynamic parameter settings.

[0017] As one possible implementation, generating a parameter dimension table based on the dynamic parameters and historical parameter values ​​includes the following steps:

[0018] Extract all data corresponding to the dynamic parameters and their historical values ​​to obtain the dynamic parameters. The dynamic parameters include date dynamic parameters and non-date dynamic parameters. Generate a unique identifier based on each combination of historical values ​​of the non-date dynamic parameters.

[0019] Each historical value combination and unique identifier of the non-date dynamic parameter is used as a field in the parameter dimension table.

[0020] As one possible implementation, the process of scheduling and executing the relational projection SQL to obtain the execution result, and then transforming the execution result into the result corresponding to the query instance, includes the following steps:

[0021] Retrieve relational projection SQL, combine data with the same identifier, and transform them into the results corresponding to the query instance SQL.

[0022] As one possible implementation, the query determines whether the result corresponding to the instance to be queried exists in the HBase database. If so, a request is made to the HBase database, including the following steps.

[0023] In response to a query request for an instance to be queried, determine whether the result corresponding to the instance to be queried exists in the HBase database; if it exists, query the HBase database directly; if it does not exist, send the request directly to the original database for data querying.

[0024] As one possible implementation, storing the data in an HBase database includes the following steps:

[0025] Store the unique identifier, the query instance SQL, as the key, and the data set with compose_uk set to 01 as the value in the key-value store HBase.

[0026] As one possible implementation, rewriting the initial SQL query pattern and parameter dimension table according to a preset model further includes the following steps: if the initial SQL query pattern contains nested SQL subqueries, then find the nearest common parent node SQL subquery where all non-date dynamic parameter dynamic parameters are located.

[0027] The SQL subquery based on the nearest common parent node is rewritten according to the preset model, using the position and parameter dimension table.

[0028] A query acceleration system based on SQL query template extraction includes a template extraction module, a parameter unique dimension table generation module, a rewrite generation module, a scheduling execution module, and a query request module.

[0029] The template extraction module extracts the initial SQL query pattern of the query instance based on historical behavior data, wherein the initial SQL query pattern includes dynamic parameters and historical values ​​of the parameters;

[0030] The parameter dimension table generation module generates a parameter dimension table based on the dynamic parameters and historical parameter values.

[0031] The rewrite generation module is used to rewrite the initial SQL query pattern and the parameter dimension table according to a preset model to generate relational projection SQL.

[0032] The scheduling and execution module is used to schedule the relational projection SQL according to the date parameter, schedule and execute the relational projection SQL to obtain the execution result, convert the execution result into all query instances corresponding to the query mode, and store the query instances and corresponding results in the HBase database in a one-to-one mapping.

[0033] The query request module is used to check whether the result corresponding to the instance to be queried exists in the HBase database. If so, it requests the HBase database.

[0034] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0035] A query acceleration device for extracting data based on an SQL query template includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the method described above.

[0036] This invention, by adopting the above technical solutions, has significant technical effects:

[0037] This invention utilizes SQL query templates and historical parameter values ​​to materialize and rewrite SQL, and schedules queries by date. It can pre-calculate all SQL instances that may be queried, thereby achieving ultimate query performance. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is a schematic diagram of the overall process of the method of the present invention;

[0040] Figure 2 This is a schematic diagram of the overall structure of the system of the present invention. Detailed Implementation

[0041] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0042] Example 1:

[0043] A query acceleration method based on SQL query templates for extracting data, such as Figure 1 As shown, it includes the following steps:

[0044] S100. Extract the initial SQL query pattern of the query instance based on historical behavior data, wherein the initial SQL query pattern includes dynamic parameters and historical values ​​of the parameters;

[0045] S200. Generate a parameter dimension table based on dynamic parameters and historical parameter values;

[0046] S300: Rewrite the initial SQL query pattern and parameter dimension table according to the preset model to generate relational projection SQL;

[0047] S400: Based on dynamic parameter scheduling relationship projection SQL, schedule execution relationship projection SQL to obtain execution results, convert the execution results into all query instances corresponding to the query mode, and store the query instances and corresponding results in the HBase database in a one-to-one mapping.

[0048] S500: Check if the result corresponding to the instance to be queried exists in the HBase database. If so, request the HBase database.

[0049] The query template is determined by the SQL query scenario. Personal information isn't actually fixed; it's just a dynamic parameter, which can be omitted here. To explain further, let's look at an example. Imagine a report with charts (each chart corresponds to a SQL query). It has dropdown menus (allowing multiple or single selection) and date selection (date is also a dropdown). It can default to displaying the latest day's data, or display different data for different people on the same page. In this case, "person" is also a dynamic field. However, the overall content displayed in the report is fixed. Essentially, the query mode (query template) is fixed, but some dynamic parameters change, such as the personnel, date, or viewing range (e.g., viewing data for the entire Hangzhou city area, or only viewing data for the Xihu district). Since the report needs to be accessed daily, the date is also dynamic.

[0050] In one embodiment, extracting the initial SQL query pattern of a query instance based on historical behavior data includes the following steps:

[0051] Retrieve historical behavior data, which includes all data for the query instance;

[0052] Retrieve all data for the query instance, including the query instance itself, and infer dynamic parameters and historical combinations of dynamic parameters based on the query instance;

[0053] All data is clustered and extracted according to a preset clustering model to obtain the corresponding initial SQL query patterns. The preset clustering model is based on dynamic parameter settings. Here, there are only two types of dynamically changing parameters: date type and other types (such as user ID, business type, etc.). For example, a chart dropdown is a dynamic parameter, which can be arbitrary. Date type parameters are used to schedule relational projection SQL to achieve the purpose of pre-executing and caching results. Non-date type parameters refer to the same query pattern, such as a chart, but where each person sees a different range of data. Here, each person's condition is a dynamic parameter. The relational projection task corresponding to the query pattern can retrieve all the data needed by everyone at once during daily scheduling, and then map it to the SQL query instance for each person based on parameter combinations, thus achieving the purpose of pre-caching.

[0054] In step S200, a parameter dimension table is generated based on dynamic parameters and historical parameter values, including the following steps:

[0055] Extract all data corresponding to the dynamic parameters and their historical values ​​to obtain the dynamic parameters. The dynamic parameters include date-based dynamic parameters and non-date-based dynamic parameters. Generate a unique identifier based on each combination of historical values ​​of the non-date-based dynamic parameters.

[0056] Each historical value combination and unique identifier of the non-date dynamic parameters is used as a field in the parameter dimension table. This table includes both date-based and non-date-based dynamic parameter types. For each combination of non-date-based dynamic parameters, a unique identifier is also generated to represent a query instance SQL.

[0057] In one embodiment, the scheduling execution relation projection SQL obtains the execution result, and the execution result is transformed into the result corresponding to the query instance, including the following steps:

[0058] Retrieve relational projection SQL, combine data with the same identifier, and transform them into the results corresponding to the query instance SQL.

[0059] In one embodiment, the query checks whether the result corresponding to the instance to be queried exists in the HBase database. If it does, the HBase database is requested, including the following steps.

[0060] In response to a query request for an instance to be queried, determine whether the result corresponding to the instance to be queried exists in the HBase database; if it exists, query the HBase database directly; if it does not exist, send the request directly to the original database for data querying.

[0061] In one embodiment, storing the data in an HBase database includes the following steps:

[0062] Store the unique identifier, the query instance SQL, as the key, and the data set with compose_uk set to 01 as the value in the key-value store HBase.

[0063] The following examples can be used to illustrate this:

[0064] In a certain banking institution, each account manager needs to query the sales data of their respective area of ​​responsibility (including team members and business types) every day. For example, some account managers' query examples are as follows:

[0065]

[0066] By clustering and extracting historical SQL query behaviors, the following query template can be obtained:

[0067]

[0068] Here, ${param1} and ${biz_param2} are dynamic parameters that can change (which can be understood as drop-down boxes or checkboxes for the corresponding report; different people may see different ranges of data for the same report).

[0069] DateTime is the date parameter. It is assumed that the report will view yesterday's data by default (of course, the time period may be weekly or monthly).

[0070] In the example above, there are multiple query instances, meaning the dynamic parameters user_id and biz_type are variable. Assume there are three possible combinations:

[0071] (1) user_id:01,02.biz_type:fund1

[0072] (2) user_id:01,03,04.biz_type: Fund 2

[0073] (3) user_id:05biz_type: Fund 3

[0074] The generated parameter unique dimension table structure:

[0075] TABLE dimension_template1{

[0076] compose_uk,

[0077] user_id,

[0078] biz_type

[0079] }

[0080] Explanation of the unique dimension table structure for parameters:

[0081] compose_uk is a unique key for the combination of dynamic parameters for each query instance. For example, the three query instances mentioned above will each generate their own combined uk value. The advantage of doing this is that a query can be restored by uk and applied to materialized SQL. This ensures that the results of materialized SQL do not require secondary calculations. The query instance can be restored simply by using uk to aggregate the result data.

[0082] The data content of the parameter unique dimension table is shown in the table below:

[0083]

[0084]

[0085] Of course, funds can also be multi-valued, with no limit on the number of dynamic parameters.

[0086] The current version's dynamic parameter combinations are derived from historical query behavior. Since a report is viewed repeatedly, calculating only those historically queried data improves pre-calculation efficiency. If faster report processing is needed, all values ​​of the dynamic parameter fields in the original table can be queried and used as data content for the parameter dimension table. Based on the template SQL and parameter dimension table, all dynamic parameter operators are rewritten to `IN` (query pushdown).

[0087] Let's continue with the example from step one. The template is:

[0088]

[0089] The parameter-only dimension table is: dimension_temnplate01 (columns: compose_uk, param1, param2).

[0090] The rewritten materialization SQL is then:

[0091]

[0092]

[0093] If the current scheduled date is 20220102, then the generated materialized SQL scheduling instance is:

[0094] Based on the results of the materialized instance SQL execution in the above steps, please refer to the following table:

[0095]

[0096]

[0097] The data from other columns of the same dimension_template01.uk file are placed into a collection. For each uk file, the parameter combination values ​​are used to reconstruct a query instance SQL. For example, if compose_uk is 01, representing the parameter combination: param1: 01, 02; param2: Fund 1, then the query instance SQL would be:

[0098]

[0099] The query instance SQL is used as the key, and the data set with compose_uk set to 0 or 1 is used as the value to store it in the key-value store HBase.

[0100] The query example above, where uk is 01, yields the following results:

[0101] user_id,count(col_1),sum(col_2) 01 100 1000 02 101 2000

[0104] When a query SQL is received, it is first checked whether the SQL (after removing spaces and other meaningless operation symbols) exists in HBase. If it exists, it is queried directly in HBase. If it does not exist, it is sent to the original SQL query engine.

[0105] In addition, the entire process includes the following steps:

[0106] If the initial SQL query pattern contains nested SQL subqueries, then find the nearest common parent node SQL subquery containing all non-date dynamic parameter dynamic parameters;

[0107] The SQL subquery based on the nearest common parent node is rewritten according to the preset model, using the position and parameter dimension table.

[0108] In this embodiment, regardless of the depth of nesting, processing starts from the innermost dynamic parameter layer. If it has not reached the position where it can be joined with the parameter dimension table, the dynamic parameter is added to the select field (if there is a group by, it also needs to be added to the group by). The relationship of parameter column passing is recorded. At the position where it can be joined with the parameter dimension table, the association relationship with each dynamic parameter in the dimension table is uniformly established.

[0109] The following are the locations where joins can be performed with parameter dimension tables:

[0110] Processing proceeds layer by layer from the innermost layer outwards. If the current layer has already processed all dynamic parameters (meaning the current layer can reference all dynamic parameters (the outer layer has no more dynamic parameters), then the current layer represents the position of the join parameter dimension table.

[0111] Summary: Find the nearest common parent node containing all dynamic parameters across all dimensions.

[0112] For example:

[0113] The template SQL is:

[0114]

[0115] By traversing the template SQL, it was found that the template SQL has two levels of queries. The innermost query has a dynamic parameter param3, and the outermost query has dynamic parameters param1 and param2. Starting from the innermost level, the dynamic parameter columns are added to the SELECT statement (if the subquery has a GROUP BY clause, then the GROUP BY clause also needs to be added). After reaching the outermost level, it was found that all dynamic parameters have been traversed. Therefore, operations on the join parameter dimension tables can be performed at the current level. The rewritten materialized SQL is as follows:

[0116] However, this materialization rewriting scheme also has some constraints. In actual practice, no templates that do not meet the requirements of materialization rewriting have been found. The constraints are as follows:

[0117] The WHERE expression containing dynamic parameters cannot contain functions; for example, `substr(col1) = xx` is not allowed.

[0118] In the WHERE expression containing dynamic parameters, the relationship between them and other expressions can only be AND, not OR. That is, dynamic parameters have a decisive relationship with the rows of the SQL query results.

[0119] Multi-value dynamic parameter position constraints: If it appears in a subquery and there are dynamic parameters in the outer layer, a template structure check is required: If the subquery is an aggregation, the outer layer must also be an aggregation, and the inner aggregation must only have dimension columns and no aggregation columns such as SUM (because if multi-value is changed to single value and appears after the select, the aggregation granularity will become smaller).

[0120] Example 2:

[0121] A query acceleration system based on SQL query templates for extracting data, such as Figure 2As shown, it includes a template extraction module 100, a parameter unique dimension table generation module 200, a rewrite generation module 300, a scheduling execution module 400, and a query request module 500;

[0122] The template extraction module 100 extracts the initial SQL query pattern of the query instance based on historical behavior data. The initial SQL query pattern includes dynamic parameters and historical values ​​of the parameters.

[0123] The parameter dimension table generation module 200 generates a parameter dimension table based on dynamic parameters and historical parameter values.

[0124] The rewrite generation module 300 is used to rewrite the initial SQL query pattern and parameter dimension table according to the preset model to generate relational projection SQL.

[0125] The scheduling and execution module 400 is used to schedule the projection SQL based on the date parameter, obtain the execution result of the projection SQL, convert the execution result into all query instances corresponding to the query mode, and store the query instances and corresponding results in the HBase database in a one-to-one mapping.

[0126] The query request module 500 is used to check whether the result corresponding to the query instance exists in the HBase database. If so, it requests the HBase database.

[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0128] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] It should be noted that:

[0133] The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0134] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A query acceleration method based on SQL query template for extracting data, characterized in that, Includes the following steps: The initial SQL query pattern for extracting query instances based on historical behavior data includes dynamic parameters and historical parameter values. A parameter dimension table is generated based on the dynamic parameters and historical parameter values; The initial SQL query pattern and the parameter dimension table are rewritten according to a preset model to generate relational projection SQL; Based on dynamic parameters, the relational projection SQL is scheduled and executed to obtain the execution result. The execution result is converted into all query instances corresponding to the query mode, and the query instances and their corresponding results are mapped one-to-one and stored in the HBase database. Check if the result corresponding to the instance to be queried exists in the HBase database. If it does, then request the HBase database.

2. The query acceleration method for extracting data based on SQL query templates according to claim 1, characterized in that, The initial SQL query pattern for extracting query instances based on historical behavior data includes the following steps: Obtain historical behavior data, which includes all data of the query instance; Obtain all data of the query instance, wherein the all data includes the query instance, and infer dynamic parameters and historical combinations of dynamic parameters based on the query instance; All data are clustered and extracted according to a preset clustering model to obtain the corresponding initial SQL query pattern. The preset clustering model is based on dynamic parameter settings.

3. The query acceleration method for extracting data based on SQL query templates according to claim 1, characterized in that, The process of generating a parameter dimension table based on the dynamic parameters and historical parameter values ​​includes the following steps: Extract all data corresponding to the dynamic parameters and their historical values ​​to obtain the dynamic parameters. The dynamic parameters include date dynamic parameters and non-date dynamic parameters. Generate a unique identifier based on each combination of historical values ​​of the non-date dynamic parameters. Each historical value combination and unique identifier of the non-date dynamic parameter is used as a field in the parameter dimension table.

4. The query acceleration method for extracting data based on SQL query templates according to claim 1, characterized in that, The process of scheduling and executing the relational projection SQL to obtain the execution result, and then transforming the execution result into the result corresponding to the query instance, includes the following steps: Retrieve relational projection SQL, combine data with the same identifier, and transform them into the results corresponding to the query instance SQL.

5. The query acceleration method for extracting data based on SQL query templates according to claim 1, characterized in that, If the query query does not contain the result of the instance to be queried, then the HBase database is requested, including the following steps; In response to a query request for an instance to be queried, determine whether the result corresponding to the instance to be queried exists in the HBase database; if it exists, query the HBase database directly; if it does not exist, send the request directly to the original database for data querying.

6. The query acceleration method for extracting data based on SQL query templates according to claim 1, characterized in that, The data stored in the HBase database includes the following steps: Combine the unique identifier with the query instance SQL as the key and the data set with compose_uk as 01 as the value and store it in the key-value store HBase.

7. The query acceleration method for extracting data based on SQL query templates according to claim 1, characterized in that, The rewriting of the initial SQL query pattern and parameter dimension table according to the preset model also includes the following steps: if the initial SQL query pattern has nested SQL subqueries, then find the nearest common parent node SQL subquery where all non-date dynamic parameters are located. The SQL subquery based on the nearest common parent node is rewritten according to the preset model, using the position and parameter dimension table.

8. A query acceleration system based on SQL query templates for extracting data, characterized in that, It includes a template extraction module, a parameter unique dimension table generation module, a rewrite generation module, a scheduling and execution module, and a query request module; The template extraction module extracts the initial SQL query pattern of the query instance based on historical behavior data, wherein the initial SQL query pattern includes dynamic parameters and historical values ​​of the parameters; The parameter dimension table generation module generates a parameter dimension table based on the dynamic parameters and historical parameter values. The rewrite generation module is used to rewrite the initial SQL query pattern and the parameter dimension table according to a preset model to generate relational projection SQL. The scheduling and execution module is used to schedule the relational projection SQL according to the date parameter, schedule and execute the relational projection SQL to obtain the execution result, convert the execution result into all query instances corresponding to the query mode, and store the query instances and corresponding results in the HBase database in a one-to-one mapping. The query request module is used to check whether the result corresponding to the instance to be queried exists in the HBase database. If so, it requests the HBase database.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

10. A query acceleration device for extracting data based on an SQL query template, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

Citation Information

Patent Citations

  • Database parameter synchronization method and system

    CN111444281A

  • Dimension data processing method and device, storage medium and electronic equipment

    CN114265887A