Data query method and device, electronic equipment and computer readable storage medium
By constructing a knowledge graph and semantic alignment to generate a global business entity view, and selecting the optimal SQL statement, the problems of ETL latency and federated query accuracy are solved, enabling real-time and efficient querying of multi-source data.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 武汉达梦数据技术有限公司
- Filing Date
- 2025-11-21
- Publication Date
- 2026-07-21
AI Technical Summary
In existing technologies, ETL latency cannot meet the needs of real-time data analysis, and federated query engines cannot perceive the new data distribution in AI training, resulting in poor data query accuracy.
By crawling data from N data sources, constructing a knowledge graph, performing semantic alignment, generating a global business entity view, selecting the optimal SQL statement for data querying, and optimizing the query plan using a reinforcement learning model.
It enables real-time data querying across multiple heterogeneous data sources, improving query accuracy and efficiency, reducing data transmission volume, and optimizing query plans.
Smart Images
Figure CN121579520B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and data query technology, and in particular to a data query method, apparatus, electronic device and computer-readable storage medium. Background Technology
[0002] As enterprises delve deeper into digitalization, data is scattered across relational databases (MySQL, PostgreSQL), NoSQL databases (MongoDB, Cassandra), data lakes (S3 + Parquet / ORC), and even streaming systems (Kafka). Related technologies employ ETL to aggregate data into a unified repository before querying; or they use federated query engines (Presto / Trino) to manually write SQL queries for different data sources and then manually optimize them.
[0003] However, ETL latency of T+1 or longer cannot meet the needs of real-time data analysis; the cost-based model (CBO) of the federated engine cannot perceive the new data distribution during AI training, resulting in suboptimal query plans and thus poor accuracy in data queries. Therefore, a new data query method is urgently needed to solve these problems. Summary of the Invention
[0004] In view of this, this application provides a data query method, apparatus, electronic device, and computer-readable storage medium, which can realize real-time data query under multiple heterogeneous data sources and has a high query accuracy.
[0005] A first aspect of this application provides a data query method, comprising: crawling data from N data sources to obtain database table fields, database table query counts, and sampled data for each data source, where N is an integer greater than 1; constructing a knowledge graph corresponding to each data source based on the database table fields, database table query counts, and sampled data; performing semantic alignment on the N knowledge graphs to obtain a global business entity view after clustering the database table fields; generating M candidate SQL statements corresponding to a natural language question to be queried based on the global business entity view, where M is an integer greater than 1; determining the query features corresponding to each candidate SQL statement, and selecting the optimal SQL statement from the M candidate SQL statements based on the query features; and performing a data query on the N data sources based on the optimal SQL statement.
[0006] In one possible implementation, before determining the query features corresponding to each candidate SQL statement, the method further includes: inputting all M candidate SQL statements into a reinforcement learning model to obtain a preferred SQL statement output by the reinforcement learning model, wherein the reinforcement learning model uses execution cost as a reward; determining the query features corresponding to each candidate SQL statement and selecting the optimal SQL statement from the M candidate SQL statements based on the query features includes: determining the query features corresponding to each preferred SQL statement; and selecting the optimal SQL statement from the preferred SQL statements based on the query features.
[0007] In one possible implementation, the query features include a filter column, an aggregation column, a join column, and the amount of query data. The step of selecting the optimal SQL statement from the preferred SQL statements based on the query features includes: inputting the filter column, the aggregation column, the join column, the amount of query data, and the number of database / table queries into a cost prediction network model to obtain the estimated time and cost of querying each preferred SQL statement in each of the data sources; calculating a score for each preferred SQL statement based on the estimated time and cost; and selecting the preferred SQL statement with the highest score as the optimal SQL statement.
[0008] In one possible implementation, calculating the score for each preferred SQL statement based on the estimated time and the cost includes: calculating the score according to the following formula:
[0009] ;in, For the stated fraction, For the estimated time, For the aforementioned cost, The cache hit rate is used to represent the number of query hits in the in-memory database. , as well as All of these are preset weights.
[0010] In one possible implementation, after crawling data from the N data sources respectively, the method further includes: obtaining the resource storage capacity of the storage hardware corresponding to each data source; the step of querying data from the N data sources according to the optimal SQL statement includes: splitting the optimal SQL statement into multiple sub-SQL statements according to the resource storage capacity; and querying data from the N data sources according to the multiple sub-SQL statements.
[0011] In one possible implementation, the step of semantically aligning the N knowledge graphs to obtain a global business entity view after clustering the database table fields includes: vectorizing the database table fields, the number of database table queries, and the data source sampling data; and automatically matching synonym / near-synonymous fields in the vectorized database table fields to generate the global business entity view.
[0012] In one possible implementation, before constructing the knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the data source sampling data, the method further includes: determining nested fields in the database table fields; flattening the nested fields to obtain multiple sub-fields corresponding to the nested fields; the step of constructing the knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the data source sampling data includes: constructing the knowledge graph based on the non-nested fields in the database table fields, the multiple sub-fields, the number of database table queries, and the data source sampling data.
[0013] Secondly, embodiments of this application also provide a data query device, including: a crawling module, a construction module, a clustering module, a generation module, a filtering module, and a query module; the crawling module is used to crawl data from N data sources respectively, to obtain the database table fields, database table query counts, and data source sampling data corresponding to each data source, where N is an integer greater than 1; the construction module is used to construct a knowledge graph corresponding to each data source based on the database table fields, the database table query counts, and the data source sampling data; the clustering module is used to perform semantic alignment on the N knowledge graphs to obtain a global business entity view after clustering the database table fields; the generation module is used to generate M candidate SQL statements corresponding to the natural language question to be queried based on the global business entity view, where M is an integer greater than 1; the filtering module is used to determine the query features corresponding to each candidate SQL statement, and to filter out the optimal SQL statement from the M candidate SQL statements based on the query features; the query module is used to perform data querying in the N data sources based on the optimal SQL statement.
[0014] Thirdly, embodiments of this application also provide an electronic device, the electronic device including a processor and a memory, the memory being used to store instructions, and the processor being used to call the instructions in the memory, causing the electronic device to execute the data query method as described in the first aspect.
[0015] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the data query method as described in the first aspect.
[0016] Compared with related technologies, the embodiments of this application have at least the following advantages: By crawling data from N data sources, the database table fields, database table query counts, and data source sampling data of each data source can be obtained, thus constructing a knowledge graph for each data source. This allows the fragmented information within the data source to be converted into a logically connected knowledge network. Then, semantic alignment is performed on the obtained N knowledge graphs to obtain a global business entity view after clustering the database table fields. This achieves data integration of the knowledge graphs, enabling the generation of M candidate SQL statements corresponding to the natural language question to be queried more quickly based on the global business entity view, thereby improving the efficiency of subsequent data queries. Furthermore, the query features corresponding to each candidate SQL statement are determined, and the optimal SQL statement is selected from the M candidate SQL statements based on the query features. Data queries are then performed on the N data sources based on the optimal SQL statement, improving the accuracy of data queries.
[0017] The technical effects achieved by the second, third, and fourth aspects mentioned above are similar to those achieved by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description
[0018] Figure 1 A flowchart illustrating the steps of a data query method provided in an embodiment of this application;
[0019] Figure 2 A flowchart illustrating another step of the data query method provided in an embodiment of this application;
[0020] Figure 3 A functional block diagram of a data query device provided in an embodiment of this application;
[0021] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0022] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.
[0023] The following description sets forth many specific details to provide a full understanding of this application. The described embodiments are only some, not all, of the embodiments of this application.
[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the specification of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application.
[0025] It should be further noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0026] In this application, "at least one" means one or more, and "more than one" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and drawings of this application are used to distinguish similar objects, not to describe a specific order or sequence.
[0027] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.
[0028] For ease of understanding, some concepts related to the embodiments of this application are illustrated and explained by way of example for reference.
[0029] SQL statements are a standard computer language used to manage and manipulate relational databases. They allow users to perform the following core operations: 1. Data Query: Retrieving data from the database using the SELECT statement; 2. Data Manipulation: Inserting, updating, and deleting data; 3. Data Definition: Creating, modifying, and deleting database structures; 4. Data Control: Managing access permissions (GRANT / REVOKE).
[0030] Knowledge graph: A semantic network that organizes knowledge in the form of a graph structure, describing real-world concepts and their relationships through elements such as entities, relations, and attributes.
[0031] Sentence-BERT is an improved method based on the pre-trained BERT model. It generates semantically sensitive sentence vector representations through Siamese or Triplet network structures, and is mainly used to solve the efficiency problem of tasks such as sentence similarity calculation, clustering, and information retrieval.
[0032] Cost prediction network models are mathematical functions used in machine learning to measure the prediction error of a model and optimize its parameters. Their core objective is to minimize the difference between predicted and true values. By quantifying the accuracy of model predictions, the cost function helps determine the optimal solution for model parameters. For example, in neural network training, cost functions (such as mean squared error or cross-entropy loss) are used to evaluate the deviation between predicted values and true labels and to optimize parameters using gradient descent.
[0033] Reinforcement learning is a machine learning framework based on Markov decision processes. It enables an agent to learn optimal policies through trial and error in interactions with the environment, maximizing long-term cumulative rewards. This model consists of core elements such as states, actions, rewards, policies, and value functions, and is widely used in fields such as games, robot control, and autonomous driving.
[0034] Time series forecasting models are mathematical models or algorithms that predict future trends by analyzing historical time series data (observations arranged in chronological order). Their core principle is to leverage the continuity and potential patterns (such as trends, periodicity, and seasonality) of data over time, combined with statistical or machine learning methods, to establish a predictive framework.
[0035] AI (Artificial Intelligence): Simulates human intelligence through computer systems, including abilities such as learning, reasoning, problem-solving, perception, and language comprehension.
[0036] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating the steps of one embodiment of the data query method of this application. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted.
[0037] It should be noted that the data query method of this application embodiment can be applied to scenarios where data queries are performed under multiple heterogeneous data sources. The executing entity can be a data query device; for example, when a user needs to perform a data query, the data query device can be used to perform data queries across multiple heterogeneous data sources. Of course, the data query method of this application embodiment can also be applied to other scenarios requiring data queries, and this application does not specifically limit its application in these cases.
[0038] The specific process of this embodiment is as follows: Figure 1 As shown, it includes the following steps:
[0039] S101, crawl data from N data sources respectively, and obtain the corresponding database table fields, database table query counts and data sampled from each data source.
[0040] Specifically, N is an integer greater than 1. Data sources include, but are not limited to, relational databases, non-relational databases, and data lakes. This embodiment does not specifically limit the type of data source; AI can be used to crawl data from any type of data source.
[0041] In some embodiments, the AI crawler connects to various data sources at preset intervals, extracting database table fields, database table query counts, and data sampled from the data sources. It is understood that this embodiment does not specifically limit the preset interval, which can be set according to actual needs. For example, the preset interval can be set to 1 day, 1 week, etc.
[0042] It should be noted that each data source includes multiple database tables. In this embodiment, the database table fields are all fields in the database tables, the database table query count is the number of times the database table is queried, and the data source sampling data is the sampled data obtained after sampling the database tables. Specifically, the sampled data can be the first few records of the database table, or data obtained after randomly sampling the database table. This embodiment does not specifically limit the sampling method.
[0043] S102, construct a knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the sampled data from the data source.
[0044] Specifically, a knowledge graph of "data source - database table - database table field" is constructed based on the database table fields, the number of database table queries, and the sampled data from the data source.
[0045] In some embodiments, before constructing the knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the data source sampling data, the method further includes: determining the nested fields in the database table fields; flattening the nested fields to obtain multiple sub-fields corresponding to the nested fields; and constructing the knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the data source sampling data, including: constructing the knowledge graph based on the non-nested fields in the database table fields, multiple sub-fields, the number of database table queries, and the data source sampling data.
[0046] Specifically, after the AI crawler extracts the database table fields, the AI automatically infers the structure of the nested JSON / BSON fields in the database table and generates a flattened view. This allows the knowledge graph to fully display all the information from the data source.
[0047] In some embodiments, the AI also creates a unified transformation UDF for heterogeneous timestamps and enumeration values in the database table fields and data source sampling data. This ensures that the database table fields and data source sampling data from different types of data sources maintain consistency in the format of timestamps and enumeration values, improving the efficiency of subsequent clustering of the database table fields.
[0048] S103, perform semantic alignment on N knowledge graphs to obtain a global business entity view after clustering the database table fields.
[0049] In some embodiments, the database table fields, database table query counts, and data source sampling data are vectorized; synonym / near-synonymous fields in the vectorized database table fields are automatically matched to generate a global business entity view.
[0050] Specifically, N knowledge graphs are input into Sentence-BERT. Sentence-BERT vectorizes the database table fields, database table field comments, database table query counts, and data source sampling data, and automatically matches synonym / near-synonymous fields (such as user_id, uid, and customer_number can be clustered) to generate a global business entity view.
[0051] S104, Generate M candidate SQL statements corresponding to the natural language question to be queried based on the global business entity view.
[0052] In some embodiments, inputting a natural language question and a global business entity view into a large speech model can yield M candidate SQL statements corresponding to the natural language question.
[0053] S105, determine the query characteristics corresponding to each candidate SQL statement, and select the optimal SQL statement from the M candidate SQL statements based on the query characteristics.
[0054] It should be noted that the method for selecting the optimal SQL statement is described in detail in subsequent embodiments, and will not be repeated here to avoid repetition.
[0055] S106, perform data querying from N data sources based on the optimal SQL statement.
[0056] In some embodiments, after crawling data from N data sources respectively, the method further includes: obtaining the resource storage capacity of the storage hardware corresponding to each data source; querying data in the N data sources according to the optimal SQL statement, including: splitting the optimal SQL statement into multiple sub-SQL statements according to the resource storage capacity; and querying data in the N data sources according to the multiple sub-SQL statements.
[0057] Specifically, during the process of crawling data from the data source, the AI crawler also crawls the storage resources and capabilities of the storage hardware storing the data source, such as indexes, hard disk partitions where the data source is located, and UDFs. The AI breaks down the optimal SQL statement and generates multiple optimal pushdown sub-SQL statements based on the resource storage capabilities. Different optimal pushdown sub-SQL statements are used to query data in the corresponding data source, which can reduce the amount of data transmission by 50%-80%.
[0058] In some embodiments, the AI also utilizes a time-series prediction model to predict query hotspots for the next 24 hours and stores the corresponding data in an in-memory database in advance. This approach further improves data query efficiency when the data being queried is stored in an in-memory database.
[0059] It's worth noting that the data stored in the in-memory database is dynamically adjusted based on the hit rate. For example, if data A stored in the in-memory database is not hit within 24 hours, meaning no user queries data A, the AI will delete data A, freeing up storage space in the in-memory database.
[0060] In some embodiments, after querying data from N data sources using the optimal SQL statement, the AI returns data in standard formats such as JSON / Arrow / Avro, automatically filling in missing fields and standardizing precision. The AI also marks potential data quality issues (such as enumeration value drift or sudden increase in null values) in the query results, prompting the user to review them.
[0061] Compared with related technologies, the embodiments of this application have at least the following advantages: By crawling data from N data sources, the database table fields, database table query counts, and data source sampling data of each data source can be obtained, thus constructing a knowledge graph for each data source. This allows the fragmented information within the data source to be converted into a logically connected knowledge network. Then, semantic alignment is performed on the obtained N knowledge graphs to obtain a global business entity view after clustering the database table fields. This achieves data integration of the knowledge graphs, enabling the generation of M candidate SQL statements corresponding to the natural language question to be queried more quickly based on the global business entity view, thereby improving the efficiency of subsequent data queries. Furthermore, the query features corresponding to each candidate SQL statement are determined, and the optimal SQL statement is selected from the M candidate SQL statements based on the query features. Data queries are then performed on the N data sources based on the optimal SQL statement, improving the accuracy of data queries.
[0062] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the steps of one embodiment of the data query method of this application. Depending on different needs, the order of the steps in this flowchart can be changed, and some steps can be omitted. This data query method can be applied to the aforementioned data query device, but is not limited thereto, and this embodiment of the application does not limit it in this regard.
[0063] This embodiment is a detailed description of the foregoing embodiments, mainly illustrating how to select the optimal SQL statement. This method can further improve the accuracy of data retrieval.
[0064] The specific process of this embodiment is as follows: Figure 2 As shown, it includes the following steps:
[0065] S201, crawl data from N data sources respectively, and obtain the corresponding database table fields, database table query counts and data sampled from each data source.
[0066] S202: Construct a knowledge graph for each data source based on the database table fields, the number of database table queries, and the sampled data from the data source.
[0067] S203, perform semantic alignment on N knowledge graphs to obtain a global business entity view after clustering the database table fields.
[0068] S204, Generate M candidate SQL statements corresponding to the natural language question to be queried based on the global business entity view.
[0069] S201 to S204 of this embodiment are similar to S101 to S104 of the previous embodiment. To avoid repetition, they will not be described again here.
[0070] S205. Input all M candidate SQL statements into the reinforcement learning model to obtain the preferred SQL statement output by the reinforcement learning model.
[0071] Specifically, the reinforcement learning model uses execution cost as a reward to filter M candidate SQL statements and output the preferred SQL statement.
[0072] It's worth noting that the output preferred SQL statements are standard SQL statements, and the AI will rewrite these standard SQL statements into the format corresponding to each data source. For example, when the data source is a relational database, data queries are performed directly using standard SQL statements; when the data source is a non-relational database, the standard SQL statements are converted to AggregationPipeline for data queries; and when the data source is an S3 data lake, the standard SQL statements are converted to SparkSQL / PrestoSQL for data queries.
[0073] S206, determine the query characteristics corresponding to each preferred SQL statement.
[0074] In some embodiments, query features include filter columns, aggregation columns, join columns, and the amount of data to be queried when using a preferred SQL statement.
[0075] S207, Select the optimal SQL statement from the preferred SQL statements based on query characteristics.
[0076] In some embodiments, the filter column, aggregation column, join column, query data volume, and number of database / table queries are input into the cost prediction network model to obtain the estimated time and cost of querying each preferred SQL statement in each data source; a score is calculated for each preferred SQL statement based on the estimated time and cost; and the preferred SQL statement with the highest score is selected as the optimal SQL statement.
[0077] In some embodiments, the score is calculated according to the following formula:
[0078] ;
[0079] in, For fractions, To estimate the time required, For cost, The cache hit rate is used to represent the number of query hits in the in-memory database. , as well as All of these are preset weights.
[0080] S208, query data from N data sources based on the optimal SQL statement.
[0081] In some embodiments, the AI crawler connects to various data sources at preset intervals, extracting database table fields, database table query counts, and data sampled from the data sources. Therefore, the AI model continuously updates the knowledge graph. Specifically, the AI model in this embodiment supports A / B testing: the old and new AI models run in parallel, with the P95 latency reduction rate as the deployment threshold. In simple terms: sorting the response times of all query requests within a certain time period from smallest to largest, the value at the 95th percentile is P95. This means that 95% of requests have a response time ≤ P95, and only 5% of requests are slower than it. Compared to the average, P95 better reflects the experience of the "vast majority of users," avoiding the misleading effect of extreme long-tail values inflating the average.
[0082] Compared with related technologies, the embodiments of this application have at least the following advantages: By crawling data from N data sources, the database table fields, database table query counts, and data source sampling data of each data source can be obtained, thus constructing a knowledge graph for each data source. This allows the fragmented information within the data source to be converted into a logically connected knowledge network. Then, semantic alignment is performed on the obtained N knowledge graphs to obtain a global business entity view after clustering the database table fields. This achieves data integration of the knowledge graphs, enabling the generation of M candidate SQL statements corresponding to the natural language question to be queried more quickly based on the global business entity view, thereby improving the efficiency of subsequent data queries. Furthermore, the query features corresponding to each candidate SQL statement are determined, and the optimal SQL statement is selected from the M candidate SQL statements based on the query features. Data queries are then performed on the N data sources based on the optimal SQL statement, improving the accuracy of data queries.
[0083] Based on the same idea as the data query method in the above embodiments, this application also provides a data query device that can be used to execute the above data query method. For ease of explanation, the structural diagram of the data query device embodiment only shows the parts related to the embodiments of this application. Those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0084] like Figure 3 As shown, the data query device 30 includes a crawling module 301, a construction module 302, a clustering module 303, a generation module 304, a filtering module 305, and a query module 306. In some embodiments, the above modules can be programmable software instructions stored in memory and executable by a processor. It is understood that in other embodiments, the above modules can also be program instructions or firmware embedded in the processor.
[0085] The crawling module 301 is used to crawl data from N data sources respectively, and obtain the database table fields, database table query counts and data sample data corresponding to each data source, where N is an integer greater than 1;
[0086] Construction module 302 is used to construct a knowledge graph corresponding to each of the data sources based on the database table fields, the number of database table queries, and the data source sampling data;
[0087] Clustering module 303 is used to perform semantic alignment on N knowledge graphs to obtain a global business entity view after clustering the database table fields;
[0088] The generation module 304 is used to generate M candidate SQL statements corresponding to the natural language question to be queried based on the global business entity view, where M is an integer greater than 1;
[0089] The filtering module 305 is used to determine the query features corresponding to each candidate SQL statement, and to filter out the optimal SQL statement from the M candidate SQL statements based on the query features.
[0090] The query module 306 is used to perform data queries in N data sources based on the optimal SQL statement.
[0091] The data query device 30 provided in the above embodiments can implement the technical solutions described in the above data query method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above data query method embodiments, and will not be repeated here.
[0092] Please refer to Figure 4 , Figure 4 This is a schematic diagram of an embodiment of the electronic device of this application. In this embodiment of the invention, the electronic device 400 includes a processor 401, a memory 402, and a display 403. Figure 4 Only some components of the electronic device 400 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.
[0093] In some embodiments, processor 401 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 402 or process data, such as the data query method in this invention.
[0094] In some embodiments, processor 401 may be a single server or a group of servers. The server group may be centralized or distributed. In some embodiments, processor 401 may be local or remote. In some embodiments, processor 401 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, intranet, multi-cloud, etc., or any combination thereof.
[0095] In some embodiments, memory 402 may be an internal storage unit of electronic device 400, such as a hard disk or memory of electronic device 400. In other embodiments, memory 402 may also be an external storage device of electronic device 400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 400.
[0096] Furthermore, the memory 402 may include both internal storage units of the electronic device 400 and external storage devices. The memory 402 is used to store application software and various types of data installed on the electronic device 400.
[0097] In some embodiments, display 403 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 403 is used to display information from electronic device 400 and to display visual user applications. Components 401-403 of electronic device 400 communicate with each other via a system bus.
[0098] In one embodiment, when the processor 401 executes the data query program in the memory 402, the following steps can be implemented:
[0099] Data is crawled from N different data sources to obtain the corresponding database table fields, database table query counts, and sampled data for each data source, where N is an integer greater than 1.
[0100] Construct a knowledge graph for each data source based on the database table fields, the number of database table queries, and the data source sampling data;
[0101] Semantic alignment is performed on the N knowledge graphs to obtain a global business entity view after clustering the database table fields;
[0102] Based on the global business entity view, generate M candidate SQL statements corresponding to the natural language question to be queried, where M is an integer greater than 1;
[0103] Determine the query features corresponding to each candidate SQL statement, and select the optimal SQL statement from the M candidate SQL statements based on the query features;
[0104] Data queries are performed on N data sources based on the optimal SQL statement.
[0105] It should be understood that when the processor 401 executes the data query program in the memory 402, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.
[0106] Furthermore, this embodiment of the invention does not specifically limit the type of electronic device 400 mentioned. Electronic device 400 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the invention, electronic device 400 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).
[0107] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the data query methods provided in the above-described method embodiments.
[0108] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.
[0109] The data query method, apparatus, electronic device, and computer-readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A data query method, characterized in that, include: Data is crawled from N different data sources to obtain the corresponding database table fields, database table query counts, and sampled data for each data source, where N is an integer greater than 1. Construct a knowledge graph for each data source based on the database table fields, the number of database table queries, and the data source sampling data; Semantic alignment is performed on the N knowledge graphs to obtain a global business entity view after clustering the database table fields; Based on the global business entity view, generate M candidate SQL statements corresponding to the natural language question to be queried, where M is an integer greater than 1; Determine the query features corresponding to each candidate SQL statement, and select the optimal SQL statement from the M candidate SQL statements based on the query features; Data queries are performed on N data sources based on the optimal SQL statement; After crawling data from N different data sources, the process also includes: Obtain the resource storage capacity of the storage hardware corresponding to each of the data sources; The step of querying data from N data sources based on the optimal SQL statement includes: Based on the resource storage capacity, the optimal SQL statement is split into multiple sub-SQL statements; Data queries are performed in N data sources based on multiple sub-SQL statements; Before constructing the knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the sampled data from the data source, the process also includes: Identify the nested fields in the database table; The nested field is flattened to obtain multiple sub-fields corresponding to the nested field; The step of constructing a knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the sampled data from the data source includes: The knowledge graph is constructed based on the non-nested fields in the database table, the multiple sub-fields, the number of database table queries, and the sampled data from the data source.
2. The data query method according to claim 1, characterized in that, Before determining the query characteristics corresponding to each of the candidate SQL statements, the process also includes: The M candidate SQL statements are all input into the reinforcement learning model to obtain the preferred SQL statement output by the reinforcement learning model, wherein the reinforcement learning model uses the execution cost as a reward. The step of determining the query features corresponding to each candidate SQL statement and selecting the optimal SQL statement from the M candidate SQL statements based on the query features includes: Determine the query features corresponding to each of the preferred SQL statements; The optimal SQL statement is selected from the preferred SQL statements based on the query characteristics.
3. The data query method according to claim 2, characterized in that, The query features include filter columns, aggregation columns, join columns, and the amount of data to be queried; The step of selecting the optimal SQL statement from the preferred SQL statements based on the query characteristics includes: By inputting the filter column, the aggregation column, the join column, the amount of query data, and the number of database table queries into the cost prediction network model, the estimated time and cost of querying each of the preferred SQL statements in each of the data sources can be obtained. Calculate a score for each of the preferred SQL statements based on the estimated time and the cost; The SQL statement with the highest score is selected as the optimal SQL statement.
4. The data query method according to claim 3, characterized in that, The step of calculating the score for each preferred SQL statement based on the estimated time and the cost includes: calculating the score according to the following formula: ； in, For the stated fraction, For the estimated time, For the aforementioned cost, The cache hit rate is used to represent the number of query hits in the in-memory database. , as well as All of these are preset weights.
5. The data query method according to claim 1, characterized in that, The step of semantically aligning the N knowledge graphs to obtain a global business entity view after clustering the database table fields includes: The database table fields, the number of database table queries, and the sampled data from the data source are vectorized. The system automatically matches synonyms / near-synonyms in the vectorized database table fields to generate the global business entity view.
6. A data query device, characterized in that, include: The system includes a crawling module, a building module, a clustering module, a generation module, a filtering module, and a query module. The crawling module is used to crawl data from N data sources respectively, and obtain the database table fields, database table query counts and data sample data corresponding to each data source, where N is an integer greater than 1; The construction module is used to construct a knowledge graph corresponding to each of the data sources based on the database table fields, the number of database table queries, and the data source sampling data. The clustering module is used to perform semantic alignment on the N knowledge graphs to obtain a global business entity view after clustering the database table fields. The generation module is used to generate M candidate SQL statements corresponding to the natural language question to be queried based on the global business entity view, where M is an integer greater than 1; The filtering module is used to determine the query features corresponding to each candidate SQL statement, and to filter out the optimal SQL statement from the M candidate SQL statements based on the query features. The query module is used to perform data queries in N data sources based on the optimal SQL statement; After crawling data from N different data sources, the process also includes: Obtain the resource storage capacity of the storage hardware corresponding to each of the data sources; The step of querying data from N data sources based on the optimal SQL statement includes: Based on the resource storage capacity, the optimal SQL statement is split into multiple sub-SQL statements; Data queries are performed in N data sources based on multiple sub-SQL statements; Before constructing the knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the sampled data from the data source, the process also includes: Identify the nested fields in the database table; The nested field is flattened to obtain multiple sub-fields corresponding to the nested field; The step of constructing a knowledge graph corresponding to each data source based on the database table fields, the number of database table queries, and the sampled data from the data source includes: The knowledge graph is constructed based on the non-nested fields in the database table, the multiple sub-fields, the number of database table queries, and the sampled data from the data source.
7. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to invoke the instructions in the memory to cause the electronic device to execute the data query method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the data query method as described in any one of claims 1 to 5.