Data query method, device and equipment, medium and product

Abstract

The invention relates to the technical field of databases, and discloses a data query method and device, equipment, a medium and a product, and the method comprises the steps: obtaining a plurality of query statements for a target database; the query statement is generated by the large language model according to an input natural language; according to the relation between the statistical tables corresponding to the multiple query statements, the multiple query statements are combined, and at least one combined statement is obtained; and executing the at least one merged statement to obtain query results corresponding to the plurality of query statements. According to the scheme, the multiple query statements can be merged according to the relation between the statistical tables corresponding to the query statements, so that the query result corresponding to the query statements can be obtained by executing the merged statements, compared with direct processing of the multiple query statements, the IO frequency of the database is reduced, the IO load of the database is reduced, and the query efficiency is improved. The unnecessary interaction overhead is reduced, and the data query efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular, to a data query method, apparatus, device, medium and product. Background Art

[0002] With the rapid growth of data scale, the generation speed of massive data is accelerating day by day, and the data scale is becoming increasingly huge. Traditional data analysis methods face computational bottlenecks and are difficult to meet the requirements of real-time and in-depth analysis. Although databases can efficiently integrate and store massive data, there are certain technical thresholds, which limit their wide application.

[0003] In the related art, a method combining NL2DSL (Natural Language to Domain-Specific Language) and an index library route is adopted to process large-scale data, enabling users to query databases using natural language. However, the query efficiency of the database query language automatically translated from natural language is low, and it is easy to generate a large IO (Input Output) load on the database.

[0004] In view of this, there is an urgent need for a data query method with high computational efficiency. Summary of the Invention

[0005] In view of this, the present invention provides a data query method, which can improve the computational efficiency of data query.

[0006] In a first aspect, the present invention provides a data query method, the method comprising: obtaining a plurality of query statements for a target database; the query statements are generated by a large language model according to the input natural language; merging the plurality of query statements according to the relationships between the statistical tables operated by the plurality of query statements to obtain at least one merged statement; and executing the at least one merged statement to obtain query results corresponding to the plurality of query statements.

[0007] In an embodiment of the present application, when a large language model receives the input natural language, it will understand the natural language and generate corresponding multiple query statements; according to the relationships between the statistical tables operated by the multiple query statements, the related query statements among the multiple query statements can be merged to obtain at least one merged statement; and then by executing the merged statement, the query results corresponding to the multiple query statements can be obtained. In the above solution, after the user inputs the natural language, the large language model can directly process the natural language input by the user to obtain corresponding multiple query statements. At this time, according to the relationships between the statistical tables corresponding to the query statements, the multiple query statements are merged, so that by executing the merged statement, the query results corresponding to the query statements can be obtained. Compared with directly processing the multiple query statements, the number of I / O operations on the database is reduced, the I / O load on the database is reduced, unnecessary interaction overhead is reduced, and the data query efficiency is improved.

[0008] In an alternative embodiment, according to the relationships between the statistical tables operated by the multiple query statements, the multiple query statements are merged to obtain at least one merged statement, including: when the statistical tables operated by the multiple query statements are all the same, the filtering conditions in the multiple query statements are merged to obtain at least one merged statement.

[0009] In an embodiment of the present application, when the statistical tables operated by the multiple query statements are all the same, the filtering conditions corresponding to the query statements are merged to obtain at least one merged statement. This can avoid accessing the same statistical table multiple times and increasing unnecessary interaction overhead, thereby improving the data query efficiency.

[0010] In an alternative embodiment, according to the relationships between the statistical tables operated by the multiple query statements, the multiple query statements are merged to obtain at least one merged statement, including: when there are at least two query statements among the multiple query statements that operate on different statistical tables, the at least two query statements are concatenated with a combination operator to obtain at least one merged statement.

[0011] In an embodiment of the present application, when the statistical tables operated by the multiple query statements are all different, the query statements are concatenated with a combination operator to obtain at least one merged statement, which can simplify the data query logic, reduce the number of queries, and thus improve the data query efficiency.

[0012] In an alternative embodiment, before executing at least one merged statement, it further includes: obtaining at least one dimension attribute in the grouping clause of the at least one merged statement; filtering the at least one dimension attribute according to the occurrence positions of the at least one dimension attribute in the merged statement to determine redundant dimension attributes; and deleting the redundant dimension attributes in the at least one merged statement.

[0013] In an embodiment of the present application, by obtaining at least one dimensional attribute in the grouping clause of the merge statement and filtering the at least one dimensional attribute according to the occurrence positions of the at least one dimensional attribute in the merge statement, redundant dimensional attributes are removed. The query logic can be simplified, unnecessary data queries can be avoided, and computing resources and storage resources can be saved.

[0014] In an alternative embodiment, the method further includes: when multiple query statements correspond to at least two dimensional attributes, determining the data corresponding to the at least two dimensional attributes respectively in the query results corresponding to the multiple query statements; generating a logical view according to the data corresponding to the at least two dimensional attributes respectively; the logical view is used to display the relationship between data according to the dimensional attributes.

[0015] In an embodiment of the present application, when multiple query statements correspond to at least two dimensional attributes, the data corresponding to the at least two dimensional attributes respectively is determined in the query results corresponding to the multiple query statements, and then a logical view can be generated according to the data corresponding to the two dimensional attributes respectively. In subsequent data queries, directly querying the logical view can simplify the query logic and improve the data query efficiency.

[0016] In an alternative embodiment, after generating a logical view including the data corresponding to at least two dimensional attributes, the method further includes: obtaining the data corresponding to the at least two dimensional attributes respectively according to the update period corresponding to the logical view; when it is detected that the data corresponding to the at least two dimensional attributes respectively is different from the data in the logical view, replacing the data in the logical view with the data corresponding to the at least two dimensional attributes respectively to update the logical view.

[0017] In an embodiment of the present application, according to the update period corresponding to the logical view, target data is obtained, and then the data in the logical view corresponding to the target data is replaced with the target data to update the logical view. The real-time performance of the logical view can be ensured, and errors in data query results caused by data changes can be avoided.

[0018] In an alternative embodiment, before obtaining multiple query statements for a target database, the method further includes: obtaining the data query volume of the target database within a specified time period; selecting a target query engine from multiple query engines according to the data query volume and the query volume intervals respectively corresponding to the multiple query engines; the target query engine is used to execute the query statements.

[0019] In an embodiment of the present application, according to the data query volume of the target database within a specified time period, and according to the data query volume and the query volume intervals respectively corresponding to the multiple query engines, selecting a suitable query engine from the multiple query engines can improve the flexibility of the method.

[0020] In an alternative embodiment, the method further includes: monitoring the working metric data of the target database; and if the working metric data is abnormal, switching the current query engine of the target database to a preset backup engine.

[0021] In the embodiments of the present application, by detecting the working metric data of the target database and switching the backup engine when the working metric data is abnormal, the robustness of the method can be improved.

[0022] In a second aspect, the present invention provides a data query device, including: an acquisition module, configured to acquire a plurality of query statements for a target database; the query statements are generated by a large language model according to the input natural language; a merging module, configured to merge the plurality of query statements according to the relationship between the statistical tables operated by the plurality of query statements to obtain at least one merged statement; and an execution module, configured to execute the at least one merged statement to obtain query results corresponding to the plurality of query statements.

[0023] In a third aspect, the present invention provides a computer device, including: a memory and a processor, which are communicatively connected to each other, wherein the memory stores computer instructions, and the processor executes the computer instructions to execute the data query method according to the first aspect or any corresponding embodiment thereof.

[0024] In a fourth aspect, a computer-readable storage medium is provided, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the above data query method.

[0025] In a fifth aspect, a computer program product or a computer program is provided, including computer instructions, and the computer instructions are used to cause a computer to execute the above data query method.

[0026] The technical solutions provided in the embodiments of this specification may include the following beneficial effects:

[0027] The present application can understand the input natural language through a large language model to obtain corresponding multiple query statements, then merge the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements to obtain at least one merged statement; and then obtain query results corresponding to the multiple query statements by executing the at least one merged statement. The above solution reduces the number of I / O operations on the database, reduces the I / O load on the database, reduces unnecessary interaction overhead, and improves the efficiency of data query by acquiring multiple query statements for the target database, merging the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements, and obtaining and executing at least one query statement to obtain the corresponding query results, as compared with directly processing multiple query statements. Description of the Drawings

[0028] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0029] Figure 1 is a schematic diagram of a scenario according to an embodiment of the present invention;

[0030] Figure 2 is a schematic flowchart of a data query method according to an embodiment of the present invention

[0031] Figure 3 is a schematic flowchart of another data query method according to an embodiment of the present invention;

[0032] Figure 4 is a schematic flowchart of another data query method according to an embodiment of the present invention;

[0033] Figure 5 is a schematic diagram of the business process of an intelligent acceleration module according to an embodiment of the present invention;

[0034] Figure 6 is a structural block diagram of a data query method device according to an embodiment of the present invention;

[0035] Figure 7 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Specific Embodiments

[0036] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0037] Figure 1 is a schematic diagram of a scenario applicable to the data query method provided by an embodiment of this application. Refer to Figure 1, after the dialogue window receives the natural language input by the user, it vectorizes the natural language question to obtain a question vector. Among them, techniques such as word embedding can be used to convert the words in the natural language question into vector form representation, and then through methods such as averaging word vectors, LSTM (Long Short-Term Memory), etc., they are aggregated into a question vector.

[0038] Send the question vector to the natural language pre-trained model for preprocessing the question vector. Optionally, the natural language pre-trained module can be models such as BERT (Bidirectional Encoder Representations from Transformers), GPT-3 (Generative Pre-trained Transformer), etc. Among them, if the question vector is sent to BERT, feature extraction can be performed on the question vector; if the question vector is sent to GPT-3, the intention of the natural language question can be understood and supplementary information related to the question can be generated.

[0039] Then send the question vector processed by the model to the vector library for retrieval. Among them, the vector library contains multiple existing vectors, and each vector can correspond to a text description. Through the vector retrieval algorithm, calculate the similarity between the question vector and each vector in the vector library, and then find the top N vectors with the highest similarity from the vector library, that is, the TOP N vectors, where N is a positive integer.

[0040] Optionally, the similarity measurement method can be cosine similarity, Euclidean distance, etc.

[0041] Extract the text descriptions corresponding to the retrieved TOP N vectors, organize them into prompt words according to a certain logic, and then input the prompt words into the LLM (Large Language Models) to generate SQL (Structured Query Language) statements that meet the requirements.

[0042] Then accelerate and optimize the SQL statement to improve the efficiency of data query. Among them, the acceleration and optimization can optimize the SQL statement from multiple aspects such as engine selection, homologous merging, dimension optimization, and logical materialization to improve the response speed of the underlying database, thereby improving the efficiency of data query.

[0043] Send the accelerated and optimized SQL statement to the underlying database, execute the query operation according to the schema and data storage structure of the underlying database, and then display the query results in the dialogue window using a visualization chart.

[0044] Embodiments of the present invention combine a series of acceleration methods such as data preprocessing, dynamic processing of data models, and optimization of distributed technology algorithms to achieve rapid analysis of large-scale data, thereby realizing real-time and efficient data insight analysis. It can simplify the data analysis process, optimize resource utilization, reduce maintenance costs, and improve the accuracy of analysis results. As a result, users can obtain real-time and accurate analysis results, significantly save computing resources and storage resources, and greatly improve the efficiency of data analysis and the accuracy of insights.

[0045] According to an embodiment of the present invention, an embodiment of a data query method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. And although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0046] In an embodiment of the present invention, a data query method is provided, which can be used for the above-mentioned mobile terminals, such as mobile phones, tablets, etc. Figure 2 is a flowchart of the data query method according to an embodiment of the present invention, as Figure 2 shown, the process includes the following steps:

[0047] Step S201, obtain multiple query statements for a target database; the query statements are generated by a large language model according to the input natural language.

[0048] Among them, the query statements are generated by a large language model according to the input natural language, and one natural language statement can correspond to multiple query statements. In a specific implementation, the query statements can be SQL statements.

[0049] After the mobile terminal receives the natural language, it will send the natural language to the large language model. The large language model performs semantic understanding on the natural language, identifies the entities related to the database and the relationships in the entities from the natural language, and judges the user's query intention based on the entity and relationship information. Then, according to the query intention, query statements corresponding to the query intention are generated. Among them, the query statements are query statements for the target database.

[0050] Step S202, merge the multiple query statements according to the relationships between the statistical tables operated by the multiple query statements to obtain at least one merged statement.

[0051] In the case where the statistical tables respectively operated by the multiple query statements are all the same, it can be judged whether there are problems such as multiple similar query conditions and redundant sub-query statements in the multiple query statements. According to the problems existing in the multiple query statements, they are merged to obtain at least one merged statement to achieve optimization of the statement structure of the multiple query statements.

[0052] For example, when there are many similar query conditions in multiple query statements, the similar query conditions in the query statements can be merged to obtain a merged statement, so as to reduce the data processing burden of data query.

[0053] For another example, when there are redundant sub-query statements in multiple query statements, the sub-query can be converted into a join query to avoid duplicate queries.

[0054] Step S203, execute at least one merged statement to obtain the query results corresponding to multiple query statements.

[0055] Read data from the statistical table operated by the query statement. After the data is read, based on the conditions in the merged statement, filter the data to only leave the data rows that meet the conditions. Then extract the required column information from the filtered data rows. In this way, the query results corresponding to multiple query statements can be obtained.

[0056] In an alternative embodiment, after obtaining the data required for the merged statement, operations such as grouping and sorting can be performed on the data, and the data after the operations is the query results corresponding to multiple query statements.

[0057] In the data query method provided by the embodiments of the present invention, when the large language model receives the input natural language, it will understand the natural language and generate corresponding multiple query statements; according to the relationship between the statistical tables respectively operated by the multiple query statements, the relevant query statements in the multiple query statements can be merged to obtain at least one merged statement; then by executing the merged statement, the query results corresponding to the multiple query statements are obtained. In the above solution, after the user inputs the natural language, the large language model can directly process the natural language input by the user to obtain the corresponding multiple query statements. At this time, according to the relationship between the statistical tables corresponding to the query statements, the multiple query statements are merged, so that by executing the merged statement, the query results corresponding to the query statements can be obtained. Compared with directly processing multiple query statements, it reduces the number of I / O operations on the database, reduces the I / O load on the database, reduces unnecessary interaction overhead, and improves the efficiency of data query.

[0058] In the embodiments of the present invention, a data query method is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 3 It is a flowchart of the data query method according to the embodiments of the present invention, as Figure 3 shown, and the process includes the following steps:

[0059] Step S301, obtain multiple query statements for the target database; the query statements are generated by the large language model according to the input natural language.

[0060] Optionally, in the embodiments of the present application, after obtaining multiple query statements for the target database, the query statements can also be verified to prevent data query failures caused by incorrect query statements and improve the robustness of the method. Among them, syntax verification and semantic verification can be performed on the query statements to ensure that the query statements comply with the syntax rules and natural language requirements of the corresponding statistical tables.

[0061] In a possible implementation manner of the embodiments of the present invention, the query statement can be generated by a large language model according to the natural language input by the user once. Specifically, when the large language model generates only one query statement according to the natural language input by the user once, in order to ensure the rate of database query, the database can directly respond to the query operation of the query language; while when multiple query statements are generated according to the natural language input by the user once, if the database processes multiple query statements in sequence at this time, it is easy to cause a large IO pressure on the database. Therefore, a specified number of query statements (i.e., multiple query statements, for example, three query statements are obtained each time) can be obtained from the multiple query statements generated by the large language model according to the natural language input by the user to perform subsequent merging operations.

[0062] Step S302: Merge the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements to obtain at least one merged statement.

[0063] In a possible implementation manner, the above step S302 includes:

[0064] Step S3021: When the statistical tables operated by the multiple query statements are the same, merge the filtering conditions in the multiple query statements to obtain at least one merged statement.

[0065] The filtering conditions corresponding to each query statement can be determined according to the format of the query statement to determine the data to be queried by the query statement. Then, the filtering conditions are placed in the metrics to merge the filtering conditions and obtain a merged statement. In specific implementation, logical operators and other methods can also be used to combine the filtering conditions to obtain a merged statement.

[0066] In some alternative embodiments, if some of the multiple query statements operate on the same statistical table, only the query statements that operate on the same statistical table can be merged. The remaining query statements that operate on different statistical tables are not merged and are directly regarded as merged statements to perform subsequent steps.

[0067] In some alternative embodiments, the statistical tables respectively operated by multiple query statements are the same, and the number of statistical tables is at least one. Then, the filtering conditions can be merged according to the statistical tables. Among them, the filtering conditions for different statistical tables can be merged respectively to obtain at least one merged statement, and each merged statement corresponds to a statistical table.

[0068] In a practical application, multiple query statements include counting the number of employees with a performance rating of A and counting the total bonus of employees with a performance rating of A. Then, the two filtering conditions can be merged to obtain a merged statement.

[0069] Among them, the pseudo-code of the query statement corresponding to counting the number of employees with a performance rating of A is as follows:

[0070] SELECT

[0071] d.department_id as department_id,

[0072] d.department_name as department_name,

[0073] COUNT(DISTINCT IF(p.performance_level = 'A',e.employee_id,NULL))as a_performance_count

[0074] FROM

[0075] company.employee_performance_table p--Employee performance table

[0076] LEFT JOIN

[0077] company.department_tabled--Department table

[0078] ON p.department_id = d.department_id

[0079] WHERE

[0080] p.evaluation_date = '2024-01-01'

[0081] GROUP BY

[0082] d.department_id,d.department_name;

[0083] The pseudo-code of the query statement corresponding to the number of employees with a statistical performance level of A is as follows:

[0084] SELECT

[0085] d.department_id as department_id,

[0086] d.department_name as department_name,

[0087] SUM(IF(p.performance_level = 'A',p.bonus,0))as a_performance_total_bonus

[0088] FROM

[0089] company.employee_performance_table p--Employee performance table

[0090] LEFT JOIN

[0091] The pseudo-code of the corresponding merge statement is as follows:

[0092] SELECT

[0093] d.department_id as department_id,

[0094] d.department_name as department_name,

[0095] COUNT(DISTINCT IF(p.performance_level = 'A',e.employee_id,NULL))as a_performance_count,

[0096] SUM(IF(p.performance_level = 'A',p.bonus,0))as a_performance_total_bonus

[0097] FROM

[0098] company.employee_performance_table p--Employee performance table

[0099] LEFT JOIN

[0100] company.department_tabled--Department Table

[0101] ON p.department_id = d.department_id

[0102] WHERE

[0103] p.evaluation_date = '2024-01-01'

[0104] GROUP BY

[0105] d.department_id,d.department_name;

[0106] In some alternative embodiments, the above step S302 further includes:

[0107] Step S3022, when there are at least two query statements operating on different statistical tables among multiple query statements, concatenate the at least two query statements with a combination operator to obtain at least one merged statement.

[0108] When there are at least two query statements operating on different statistical tables among multiple query statements, the at least two query statements can be concatenated with a combination operator to obtain at least one merged statement.

[0109] In a specific implementation, the at least two query statements to be concatenated can be determined first, and then the multiple query statements can be merged in the way of union all. By executing the corresponding merged statement, data from different statistical tables but with similar structures can be merged together, facilitating subsequent analysis of the query results.

[0110] In a practical application, multiple query statements include counting the total sales amount and counting the number of page views. Then the total sales amount and the number of page views can be merged.

[0111] Among them, the pseudo code for counting the total sales amount is as follows:

[0112] SELECT SUM(amount) AS total_sales_amount FROM sales WHERE deal_flag = 1;

[0113] The pseudo code for counting the total sales amount is as follows:

[0114] SELECT COUNT(*) AS page_views FROM page_views;

[0115] The pseudo-code for combining the total sales amount and the number of page views is as follows:

[0116] SELECT SUM(amount)AS total_sales_amount,NULL AS page_views FROM sales

[0117] WHERE deal_flag=1

[0118] UNION ALL

[0119] SELECT NULL AS total_sales_amount,COUNT(*)AS page_views FROM page_views;

[0120] In some alternative embodiments, steps S3021 and S3022 may be combined and used according to the actual situation.

[0121] Among them, after obtaining multiple query statements for the target database, the query statements may first be divided according to the statistical tables operated by the multiple query statements to obtain several different groups of query statements. Each group of query statements includes several query statements, and the statistical tables operated by each query statement are the same; the statistical tables operated by different groups of query statements are different.

[0122] First, merge the query statements within the same group of query statements to obtain at least one merged statement. For details, please refer to step S2021.

[0123] Then, splice the at least one merged statement corresponding to each group of query statements using a combination operator to perform a secondary merge on the query statements and obtain at least one merged statement.

[0124] Step S303: Execute at least one merged statement to obtain the query results corresponding to the multiple query statements.

[0125] For details, please refer to Figure 2 Step S203 of the illustrated embodiment, which will not be elaborated here.

[0126] In some alternative embodiments, before step S303, it further includes:

[0127] Step a1: Obtain at least one dimensional attribute in the grouping clause of at least one merged statement.

[0128] The method of manually parsing strings may be used to extract at least one dimensional attribute of the grouping clause. In an SQL statement, the dimensional attribute is the column name listed after the SELECT keyword.

[0129] Related tools can also be used to parse the merged statement into a syntax tree to extract dimension attributes.

[0130] Step a2: Filter at least one dimension attribute according to its occurrence position in the merged statement to determine redundant dimension attributes.

[0131] Among them, business analysis can be performed on the merged statement first to obtain the analysis requirements of the merged statement. Then, it is judged whether there is a direct association between the dimension attribute and the analysis requirements. A dimension attribute that has no direct association and cannot provide valuable information for the analysis is a redundant dimension attribute.

[0132] Step a3: Delete the redundant dimension attributes in at least one merged statement.

[0133] Deleting the redundant dimension attributes to update the merged statement can make the entire data link free of data with redundant dimension attribute fields, avoiding waste. On the data service side, the data corresponding to the redundant dimension attributes is then spliced into the query result of the merged statement, which can avoid data link redundancy and a soaring number of complex query logic operations, improve query efficiency, and save computing resources and storage resources.

[0134] In specific implementation, when analyzing a query statement, there are often some dimension attribute fields without analysis requirements. When querying data, if analyzed according to this dimension attribute, this dimension attribute field needs to be added in data aggregation, which is a huge challenge to query performance. In addition, if the entire data link carries the data of this part of attribute fields, it will inevitably cause great waste. Based on this, on the premise that it is determined that no analysis is performed on this type of dimension, this part of data is not produced in the entire data link, and only the results are spliced on the data service side as needed. This can not only improve query efficiency but also save computing resources and storage resources to a great extent.

[0135] In a practical application, the application scenario of the data query method is to query the overall performance indicators of each department. The dimension attributes include department ID, department head, and department phone, etc. The department ID is closely related to the query target, and the performance indicators of different departments can be distinguished through the department ID, while the department head and department phone have no direct connection with the overall performance indicators of the department and can be used as redundant dimension attributes. After deleting the redundant dimension attributes and updating the merged statement, the query is performed. Then, according to the department ID in the query result, information such as the department head and department phone is queried and filled into the statistical result information.

[0136] Among them, the pseudo-code of the query statement showing all dimension attribute fields can be:

[0137] SELECT

[0138] d.department_id as department_id,

[0139] d.department_name as department_name,

[0140] d.department_manager as department_manager,

[0141] d.department_tel as department_tel,

[0142] COUNT(DISTINCT IF(p.performance_level = 'A',e.employee_id,NULL)) as a_performance_count,

[0143] SUM(IF(p.performance_level = 'A',p.bonus,0)) as a_performance_total_bonus FROM

[0144] company.employee_performance_table p -- Employee performance table

[0145] LEFT JOIN

[0146] company.department_table d -- Department table

[0147] ON p.department_id = d.department_id

[0148] WHERE

[0149] p.evaluation_date = '2024-01-01'

[0150] GROUP BY

[0151] d.department_id,d.department_name, d.department_manager,d.department_tel

[0152] After deleting redundant dimensional attributes, the pseudocode for the merged statement can be:

[0153] SELECT

[0154] d.department_id as department_id,

[0155] COUNT(DISTINCT IF(p.performance_level = 'A',e.employee_id,NULL))as a_performance_count,

[0156] SUM(IF(p.performance_level = 'A',p.bonus,0))as a_performance_total_bonusFROM

[0157] company.employee_performance_table p-- Employee performance table

[0158] LEFT JOIN

[0159] company.department_tabled-- Department table

[0160] ON p.department_id = d.department_id

[0161] WHERE

[0162] p.evaluation_date = '2024-01-01'

[0163] GROUP BY

[0164] d.department_id

[0165] The data query method provided by the embodiments of the present invention, when the large language model receives the input natural language, will understand the natural language and generate corresponding multiple query statements; according to the relationships between the statistical tables respectively operated by the multiple query statements, the relevant query statements in the multiple query statements can be merged to obtain at least one merged statement; then by executing the merged statement, the query results corresponding to the multiple query statements can be obtained. In the above solution, after the user inputs the natural language, the large language model can directly process the natural language input by the user to obtain the corresponding multiple query statements. At this time, according to the relationships between the statistical tables corresponding to the query statements, the multiple query statements are merged, so that by executing the merged statement, the query results corresponding to the query statements can be obtained. Compared with directly processing the multiple query statements, it reduces the number of I / O operations on the database, reduces the I / O load on the database, reduces unnecessary interaction overhead, and improves the efficiency of data query.

[0166] In this embodiment, a data query method is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablets, etc. Figure 4 It is a flowchart of the data query method according to the embodiment of the present invention, as Figure 4 shown. The process includes the following steps:

[0167] Step S401, obtain the data query volume of the target database within a specified time period.

[0168] Among them, the target database provides a query object for data query. The data query volume of the target database within a specified time period can be estimated according to the actual situation. It is also possible to use a database monitoring tool to monitor the target database to obtain the data query volume within a specified time period.

[0169] In a specific implementation, the flow-batch integration ability of DLH (Data Lake House) can be used to regularly count and manage the data volume of the target database.

[0170] Step S402, select a target query engine from multiple query engines according to the data query volume and the query volume intervals respectively corresponding to the multiple query engines; the target query engine is used to execute the query statement.

[0171] Among them, it is possible to first obtain the query volume intervals respectively corresponding to the multiple query engines, and then select a query engine that matches the data query volume from the multiple query engines as the target query engine according to the size of the data query volume.

[0172] In a specific implementation, when the connection relationship of each statistical table in the target database is multi-table association, when the data query volume is small, Presto is selected, which can efficiently perform distributed SQL processing. When the data query volume is large, Spark is selected, which is suitable for large-scale distributed computing and processing. When the connection relationship of each statistical table in the target database is single-table association, when the data query volume is small, Presto is selected to quickly return the result. When the data query volume is large, ClickHouse is selected, which is good at processing high-throughput OLAP (Online Analytical Processing) queries.

[0173] In some alternative embodiments, after step S402, it further includes:

[0174] Step b1, monitor the working index data of the target database.

[0175] The target database can be monitored through relevant monitoring tools to monitor the health status and load of the real-time monitoring engine. The working metric data can include the load status and monitoring status of the CPU (Central Processing Unit), memory, and network I / O, etc.

[0176] Step b2, if the working metric data is abnormal, switch the current query engine of the target database to a preset standby engine.

[0177] In the case where the working metric data is abnormal, immediately switch to the preset standby engine, reassign the unfinished query tasks, dynamically adjust the engine, and implement the automatic switching function of the engine to ensure the high availability and robustness of the system.

[0178] Step S403, obtain multiple query statements for the target database; the query statements are generated by the large language model according to the input natural language.

[0179] For details, please refer to Figure 2 Step S201 of the illustrated embodiment, which will not be elaborated here.

[0180] In some alternative embodiments, after step S403, it includes:

[0181] Step c1, when the multiple query statements correspond to at least two dimensional attributes, determine the data corresponding to the at least two dimensional attributes respectively in the query results corresponding to the multiple query statements.

[0182] Step c2, generate a logical view according to the data corresponding to the at least two dimensional attributes respectively; the logical view is used to display the relationship between the data according to the dimensional attributes.

[0183] When the multiple query statements correspond to at least two dimensional attributes, it is possible to first determine the data corresponding to the at least two dimensional attributes respectively in the query results corresponding to the query statements, and then analyze the at least two dimensional attributes to obtain the logical structure between the at least two dimensional attributes, where the logical structure refers to the data relationship such as the hierarchical structure, cross relationship, or dependency relationship between the data corresponding to the at least two dimensional attributes.

[0184] For example, when the dimensional attributes corresponding to the query statement include time and sales amount, the logical structure can include how the sales amount changes over time.

[0185] After obtaining the relationship between the data, it is possible to determine the data to be included in the view according to the data relationship, then read the corresponding data, and generate a logical view. The logical view can be in the form of a pie chart, bar chart, etc., to display the relationship between the data according to the dimensional attributes.

[0186] For example, when multiple query statements correspond to two dimensional attributes, namely, the employee and department dimensional attributes, the logical view can respectively display the performance of employees and the performance of departments.

[0187] In a practical application, a composite dimension is required to calculate the number of items with quarterly performance of A and the total bonus for performance of A in each quarter from sales data by production department and quarter, and a materialized view is used to materialize this logic. When querying later, the view can be directly queried.

[0188] Among them, the pseudo-code for creating a department performance view is as follows:

[0189] CREATE MATERIALIZED VIEW department_performance_summary AS SELECT

[0190] d.department_id as department_id,

[0191] d.department_name as department_name,

[0192] p.performance_quarter as performance_quarter, -- The quarter to which the performance belongs

[0193] COUNT(DISTINCT IF(p.performance_level = 'A', e.employee_id, NULL)) as a_performance_count,

[0194] SUM(IF(p.performance_level = 'A', p.bonus, 0)) as a_performance_total_bonus FROM

[0195] company.employee_performance_table p -- Employee performance table

[0196] LEFT JOIN

[0197] company.department_table d -- Department table

[0198] ON p.department_id = d.department_id

[0199] WHERE

[0200] It should be noted that in the original text, "p.performance_level='A'" has an incorrect equal sign. It should be "p.performance_level = 'A'". The above translation has been corrected accordingly.p.evaluation_date = '2024-01-01'

[0201] GROUP BY

[0202] d.department_id, d.department_name, p.performance_quarter

[0203] In some alternative embodiments, after step c2, the following steps are further included:

[0204] Step c3, according to the update period corresponding to the logical view, obtain the data corresponding to at least two dimension attributes respectively.

[0205] Among them, the update period of the data and the target data can be determined according to the actual situation. When the change frequency of at least one type of data included in the logical view is fast and the data volume is small, the update period can be shorter; when the change frequency of the data in the logical view is slow and the data volume is large, the update period can be set to be longer. At this time, the computer device can re-obtain the data (corresponding to at least two dimension attributes respectively) used to construct the logical view according to the update period corresponding to the logical view, so as to further determine whether the data used to construct the logical view has been updated. If it has been updated, the updated data needs to be synchronized to the logical view to ensure the timeliness of the logical view.

[0206] Step c4, when it is detected that the data corresponding to the at least two dimension attributes respectively is different from the data in the logical view, replace the data in the logical view with the data corresponding to the at least two dimension attributes respectively to update the logical view.

[0207] In a possible implementation manner of the embodiments of the present application, the computer device can extract the data corresponding to at least two dimension attributes respectively from the target database corresponding to the query statement. If the computer device detects that there are difference data between the data corresponding to the at least two dimension attributes respectively and the data in the logical view, directly replace the difference data corresponding to the logical view to complete the update of the logical view.

[0208] In a practical application, the pseudo code for updating the logical view in PostgreSQL is as follows:

[0209] REFRESH MATERIALIZEDVIEW department_performance_summary;

[0210] In another possible implementation, if data corresponding to at least two dimension attributes is obtained according to the update period corresponding to the logical view, and it is detected that the data corresponding to at least two dimension attributes is different from the data corresponding to the logical view, the computer device can directly use the data corresponding to at least two dimension attributes obtained in this update period to regenerate the logical view to complete the update of the logical view.

[0211] Step S404: Combine multiple query statements according to the relationships between the statistical tables operated by the multiple query statements to obtain at least one combined statement.

[0212] For details, please refer to Figure 2 Step S202 of the embodiment shown, which will not be elaborated here.

[0213] Step S405: Execute at least one combined statement to obtain the query results corresponding to the multiple query statements.

[0214] For details, please refer to Figure 2 Step S203 of the embodiment shown, which will not be elaborated here.

[0215] The data query method provided by the embodiments of the present invention, when the large language model receives the input natural language, will understand the natural language and generate corresponding multiple query statements; according to the relationships between the statistical tables respectively operated by the multiple query statements, relevant query statements in the multiple query statements can be combined to obtain at least one combined statement; and then by executing the combined statement, the query results corresponding to the multiple query statements are obtained. In the above solution, after the user inputs the natural language, the large language model can directly process the natural language input by the user to obtain the corresponding multiple query statements. At this time, according to the relationships between the statistical tables corresponding to the query statements, the multiple query statements are combined, so that by executing the combined statement, the query results corresponding to the query statements can be obtained. Compared with directly processing the multiple query statements, it reduces the number of I / O operations on the database, reduces the I / O load on the database, reduces unnecessary interaction overhead, and improves the efficiency of data query.

[0216] Figure 5 is a schematic diagram of the business process of the intelligent acceleration module shown in an exemplary embodiment. Through the Figure 5 intelligent acceleration module shown as Figures 2 to 4 the functions shown in the corresponding embodiment can be realized. As Figure 5 shown, the intelligent acceleration module is divided into an SQL preprocessing unit, an SQL query optimization unit, and an SQL result processing unit.

[0217] After the intelligent acceleration module receives the SQL statements generated by the LLM large language model, the SQL preprocessing unit will first preprocess the SQL statements from three aspects: composite dimension logic materialization, engine optimization, and attribute column pruning.

[0218] Among them, composite dimension logic materialization is based on the composite dimensions of data and materializes them into views, which can greatly improve query efficiency. In practical applications, a timing mechanism can also be configured for the views to refresh the view data regularly.

[0219] Engine optimization can select the corresponding query engine according to the actual application scenario.

[0220] Attribute column pruning can represent the deletion of dimension attributes without analysis requirements. In specific implementation, corresponding field data is not generated in the data link. Only on the data service side, the dimension attribute field data without analysis requirements and the query results of the remaining dimension attributes are concatenated.

[0221] The SQL preprocessing unit sends the preprocessed SQL statements to the SQL query optimization unit for query optimization. The SQL query optimization unit optimizes the SQL statements from two aspects: homologous model merging and using the materialized view for query in composite dimension queries.

[0222] Among them, the core concept of homologous model merging is to minimize the number of queries, which can include homologous merging and non-homologous merging. Among them, homologous merging means that the statistical tables used by multiple SQL statements are the same table. The filtering conditions can be integrated into the metrics and processed using case when (or IF). Non-homologous merging means that the statistical tables used by multiple SQL statements are different tables, and they can be merged using the union all method.

[0223] Use the SQL statements optimized by accelerated query to perform data query in the underlying database and obtain the query results. The SQL result processing unit performs attribute concatenation on the query results of the SQL statements to obtain the query results corresponding to the SQL statements. Then the query results are presented to the user on the page.

[0224] Optionally, the page presentation can display the query results through visual charts.

[0225] The intelligent acceleration module provided in this embodiment optimizes the design from perspectives such as engine optimization, homologous merging, caching strategy, and semantic optimization, so that data insight analysis can achieve good intelligent model matching, improve data query efficiency and system stability, and accelerate the performance of large-scale data processing.

[0226] In this embodiment, a data query device is further provided. This device is used to implement the above embodiments and preferred implementation manners, and those that have been described will not be elaborated again. As used hereinafter, the term "module" can be a combination of software and / or hardware that realizes a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0227] This embodiment provides a data query device, as Figure 6 shown, including:

[0228] An acquisition module 601, configured to acquire a plurality of query statements for a target database; the query statements are generated by a large language model according to the input natural language.

[0229] A merging module 602, configured to merge the plurality of query statements according to the relationship between the statistical tables operated by the plurality of query statements, to obtain at least one merged statement.

[0230] An execution module 603, configured to execute at least one merged statement to obtain query results corresponding to the plurality of query statements.

[0231] In some optional implementation manners, the merging module 602 includes:

[0232] A first merging unit, configured to merge the filtering conditions in the plurality of query statements when the statistical tables operated by the plurality of query statements are all the same, to obtain at least one merged statement.

[0233] In some optional implementation manners, the merging module 602 includes:

[0234] A second merging unit, configured to splice at least two query statements through a combination operator when there are at least two query statements among the plurality of query statements that operate on different statistical tables, to obtain at least one merged statement.

[0235] In some optional implementation manners, the data query device further includes an update module 604, configured to update the merged statement. The update module 604 includes:

[0236] A dimension attribute acquisition unit, configured to acquire at least one dimension attribute in the grouping clause of at least one merged statement;

[0237] A redundant dimension attribute determination unit, configured to screen at least one dimension attribute according to the occurrence position of at least one dimension attribute in the merged statement, to determine redundant dimension attributes;

[0238] An update unit, configured to delete the redundant dimension attributes in at least one merged statement.

[0239] In some alternative embodiments, the data query device further includes a generation module 605 for generating a logical view. The generation module 605 includes:

[0240] A data determination unit for determining, in the query results corresponding to multiple query statements, the data corresponding to at least two dimensional attributes respectively when the multiple query statements correspond to at least two dimensional attributes;

[0241] A logical view generation unit for generating a logical view according to the data corresponding to at least two dimensional attributes respectively; the logical view is used to display the relationship between data according to dimensional attributes.

[0242] In some alternative embodiments, the generation module 605 further includes:

[0243] A target data determination unit for obtaining the data corresponding to at least two dimensional attributes respectively according to the update period corresponding to the logical view.

[0244] A logical view update unit for, when detecting that the data corresponding to at least two dimensional attributes respectively is different from the data in the logical view, replacing the data in the logical view with the data corresponding to at least two dimensional attributes respectively to update the logical view.

[0245] In some alternative embodiments, the data query device includes an engine selection module 606 for selecting a query engine. The engine selection module 606 includes:

[0246] A data query volume acquisition unit for acquiring the data query volume of the target database within a specified time period;

[0247] A query engine selection unit for selecting a target query engine from multiple query engines according to the data query volume and the query volume intervals respectively corresponding to the multiple query engines; the target query engine is used to execute query statements.

[0248] In some alternative embodiments, the engine selection module 606 further includes:

[0249] A monitoring unit for monitoring the working index data of the target database;

[0250] A switching unit for, if the working index data is abnormal, switching the current query engine of the target database to a preset standby engine.

[0251] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding above embodiments and will not be elaborated here.

[0252] The data query device in this embodiment is presented in the form of a functional unit. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0253] An embodiment of the present invention further provides a computer device having the above Figure 6 shown data query device.

[0254] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of a computer device provided by an optional embodiment of the present invention. As Figure 7 shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 7 In

[0255] FIG. 16, one processor 10 is taken as an example.

[0256] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0257] The memory 20 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the computer device and the like. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 may optionally include a memory remotely disposed relative to the processor 10, and these remote memories may be connected to the computer device through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0258] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk, or a solid-state drive; the memory 20 may further include a combination of the above types of memories.

[0259] The computer device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 may be connected through a bus or other means. Figure 7 Taking connection through a bus as an example.

[0260] The input device 30 may receive input numerical or character information, and generate key signal inputs related to the user settings and function controls of the computer device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes, but is not limited to, a liquid crystal display, a light-emitting diode, a display, and a plasma display. In some alternative embodiments, the display device may be a touch screen.

[0261] Embodiments of the present invention also provide a computer-readable storage medium. The method according to the embodiments of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored as such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code, and when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the method shown in the above embodiments is implemented.

[0262] A part of the present invention can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, can call or provide the method and / or technical solution according to the present invention. Those skilled in the art should be able to understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Herein, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible by the computer.

[0263] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A data query method, characterized in that: The method comprises: Acquire multiple query statements for a target database; the query statements are generated by a large language model according to an input natural language; Merging the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements to obtain at least one merged statement; The at least one merge statement is executed to obtain query results corresponding to the multiple query statements.

2. The method according to claim 1, characterized in that The step of merging the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements to obtain at least one merged statement includes: When the statistical tables operated by the multiple query statements are all the same, the screening conditions in the multiple query statements are merged to obtain at least one merged statement.

3. The method according to claim 1, characterized in that The step of merging the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements to obtain at least one merged statement includes: When there are at least two query statements among the multiple query statements that operate on different statistical tables, the at least two query statements are concatenated through a combination operator to obtain at least one merged statement.

4. The method according to any one of claims 1 to 3, characterized in that: Before executing the at least one merge statement, the method further includes: Obtaining at least one dimension attribute in a grouping clause of the at least one merge statement; According to the appearance position of the at least one dimension attribute in the merge statement, the at least one dimension attribute is screened to determine redundant dimension attributes; The redundant dimension attributes in the at least one merge statement are deleted.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: In a case where the multiple query statements correspond to at least two dimensional attributes, determining data corresponding to the at least two dimensional attributes respectively in the query results corresponding to the multiple query statements; A logical view is generated according to the data respectively corresponding to the at least two dimensional attributes; the logical view is used to display the relationship between the data according to the dimensional attributes.

6. The method according to claim 5, characterized in that After generating the logical view including the data corresponding to the at least two dimensional attributes, the method further includes: According to the update cycle corresponding to the logical view, acquiring data corresponding to the at least two dimensional attributes respectively; When it is detected that the data respectively corresponding to the at least two dimensional attributes are different from the data in the logical view, the data respectively corresponding to the at least two dimensional attributes replace the data in the logical view to update the logical view.

7. The method according to claim 1, characterized in that Before obtaining a plurality of query statements for the target database, the method further includes: Get the data query volume of the target database within a specified time period; A target query engine is selected from the multiple query engines according to the data query volume and the query volume intervals respectively corresponding to the multiple query engines; the target query engine is used to execute the query statement.

8. The method according to claim 7, characterized in that The method further comprises: Monitoring the working indicator data of the target database; If the work indicator data is abnormal, the current query engine of the target database is switched to a preset backup engine.

9. A data query device, characterized in that: The device comprises: An acquisition module, used to acquire multiple query statements for a target database; the query statements are generated by a large language model according to an input natural language; A merging module, configured to merge the multiple query statements according to the relationship between the statistical tables operated by the multiple query statements to obtain at least one merged statement; An execution module is used to execute the at least one merge statement to obtain query results corresponding to the multiple query statements.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data query method according to any one of claims 1 to 8 by executing the computer instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data query method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the data query method according to any one of claims 1 to 8.

Images