Query system, method and equipment based on conversion from natural language to SQL (Structured Query Language) and medium

Abstract

The invention discloses a query system, method and equipment based on conversion from a natural language to an SQL (Structured Query Language) and a medium. The query system comprises a data access layer used for receiving a natural language query request of a user; the preprocessing module is used for preprocessing the natural language input by the user to obtain the preprocessed natural language; the deep learning model is used for converting the preprocessed natural language into an SQL (Structured Query Language) statement; the optimization module is used for optimizing the SQL statement to obtain an optimized SQL statement; the database interface module is used for executing the optimized SQL statements and returning query results corresponding to the optimized SQL statements; and the application layer is used for displaying the query result. It can be seen that a large amount of manual data annotation is not needed, the system development and maintenance cost is reduced, the accurate SQL statement is rapidly generated through multi-step processing and the optimization algorithm, and the efficiency of converting natural language query into the SQL statement can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of database query and natural language processing, and in particular to a query system, method, device and medium based on natural language to SQL conversion. Background Art

[0002] With the advent of the big data era, databases are increasingly being used in various fields. However, traditional database query methods require users to master complex SQL syntax, which has a high threshold for users with non-technical backgrounds. Although there are some natural language to SQL conversion methods on the market, these methods usually require a large amount of labeled data and a long machine learning process, and there are problems such as uncontrollable training time and non-reusable systems, resulting in low efficiency in converting natural language queries to SQL statements. Summary of the invention

[0003] In view of this, the present application provides a query system, method, device and medium based on natural language to SQL conversion, so as to improve the efficiency of converting natural language queries into SQL statements.

[0004] In a first aspect, the present application provides a query system based on natural language to SQL conversion, the system comprising:

[0005] Data access layer, used to receive natural language query requests from users;

[0006] A preprocessing module, used to preprocess the natural language input by the user to obtain the preprocessed natural language;

[0007] A deep learning model, used to convert the preprocessed natural language into SQL statements, the deep learning model comprising an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result;

[0008] An optimization module, used to optimize the SQL statement to obtain an optimized SQL statement;

[0009] A database interface module, used to execute the optimized SQL statement and return the query result corresponding to the optimized SQL statement;

[0010] The application layer is used to display the query results.

[0011] Optionally, the data access layer includes:

[0012] The authentication submodule is used to verify the identity of the user;

[0013] The input validation submodule is used to validate the legitimacy of the natural language input by the user using regular expressions and predefined rules;

[0014] A load balancing submodule, configured to allocate system resources to the natural language query request using a container orchestration tool;

[0015] The logging submodule is used to log the natural language query request.

[0016] Optionally, the preprocessing module includes:

[0017] A word segmentation submodule, used for using a word segmenter to segment the natural language to obtain the natural language after word segmentation;

[0018] A part-of-speech tagging submodule is used to perform part-of-speech tagging on the natural language after word segmentation to obtain the tagged natural language;

[0019] An entity recognition submodule, used to perform entity recognition on the annotated natural language to obtain recognized natural language;

[0020] A stop word removal submodule, used to remove stop words from the recognized natural language to obtain the natural language after the stop words are removed;

[0021] The conversion submodule is used to use a pre-trained word vector model to convert the natural language after the stop words are removed into a word vector as an input of the deep learning model.

[0022] Optionally, the deep learning model is a pre-trained language model based on Transformer, and the training process of the deep learning model includes:

[0023] Acquire a training data set, wherein the training data set includes a text corpus;

[0024] The deep learning model is trained using the training data set, and the model parameters of the deep learning model are optimized based on the PyTorch framework or the TensorFlow framework until the model convergence condition is reached, thereby obtaining the deep learning model.

[0025] Optionally, the optimization module includes:

[0026] A grammar checking submodule, used to check the grammar of the SQL statement;

[0027] A semantic verification submodule is used to verify whether the SQL statement conforms to the user's intention by using a machine learning model;

[0028] A query optimization submodule, used to optimize the query performance of the SQL statement using an optimizer;

[0029] The query execution plan generation submodule is used to generate a query execution plan for the SQL statement and estimate the query cost.

[0030] Optionally, the database interface module includes:

[0031] A connection pool management submodule, used to dynamically adjust the number of connections in the connection pool between the query system and the database according to the concurrency situation;

[0032] The query execution submodule is used to execute SQL queries using asynchronous execution mode;

[0033] The cache submodule is used to cache query plans and result sets.

[0034] Optionally, the application layer includes:

[0035] Query visualization submodule, used to provide real-time preview function;

[0036] A result display submodule, used to display the query results;

[0037] The permission management submodule is used to manage user access rights to different data sources and functions;

[0038] The feedback submodule is used to collect users' evaluation and improvement suggestions on the generated query results.

[0039] In a second aspect, the present application provides a natural language to SQL query method, which is applied to the system as claimed in the first aspect, and the method comprises:

[0040] receiving a natural language query request, wherein the request includes natural language input by a user;

[0041] Preprocessing the natural language to obtain preprocessed natural language;

[0042] The preprocessed natural language is converted into an SQL statement using a deep learning model, wherein the deep learning model includes an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result;

[0043] Optimizing the SQL statement to obtain an optimized SQL statement;

[0044] Execute the optimized SQL statement and return the query result corresponding to the optimized SQL statement;

[0045] The query result is displayed.

[0046] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0047] A memory for storing one or more programs;

[0048] The processor implements the query method as described in the second aspect when the one or more programs are executed by the processor.

[0049] In a fourth aspect, an embodiment of the present application provides a computer storage medium, in which a program is stored. When the program is executed by a processor, the query method described in the second aspect is implemented.

[0050] The above technical solution has the following beneficial effects:

[0051] The embodiment of the present application provides a query system, method, device and medium based on natural language to SQL conversion, the query system includes: a data access layer for receiving a user's natural language query request; a preprocessing module for preprocessing the natural language input by the user to obtain the preprocessed natural language; a deep learning model for converting the preprocessed natural language into an SQL statement, the deep learning model includes an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result; an optimization module for optimizing the SQL statement to obtain an optimized SQL statement; a database interface module for executing the optimized SQL statement and returning the query result corresponding to the optimized SQL statement; an application layer for displaying the query result. It can be seen that the present application does not require a large amount of manually annotated data, reduces the system development and maintenance costs, and quickly generates accurate SQL statements through multi-step processing and optimization algorithms, which can improve the efficiency of converting natural language queries into SQL statements. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A schematic diagram of a structure of a query system based on natural language conversion to SQL provided in an embodiment of the present application;

[0054] Figure 2 A flowchart of a natural language to SQL query method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0056] In order to facilitate further understanding of the technical solution provided by this application, the background technology involved in this application will be explained below.

[0057] Traditional database query methods require users to have professional technical knowledge, especially the need to master the complex SQL (Structured Query Language) syntax. SQL is a standard programming language used to interact with, query, update and manage database data. However, for users without technical background or professional training, learning and mastering SQL syntax may be quite difficult, which constitutes a high threshold for use. In other words, this traditional method is not friendly enough for non-technical users to directly use the database for query.

[0058] Although there are some natural language to SQL conversion methods on the market, these methods usually require a large amount of labeled data and a long machine learning process, and there are problems such as uncontrollable training time and non-reusable systems.

[0059] In order to overcome the above technical problems, the embodiment of the present application provides a query system, method, device and medium based on natural language to SQL, the query system includes: a data access layer, used to receive the user's natural language query request; a preprocessing module, used to preprocess the natural language input by the user to obtain the preprocessed natural language; a deep learning model, used to convert the preprocessed natural language into an SQL statement, the deep learning model includes an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain the corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result; an optimization module, used to optimize the SQL statement to obtain an optimized SQL statement; a database interface module, used to execute the optimized SQL statement, and return the query result corresponding to the optimized SQL statement; an application layer, used to display the query result. It can be seen that the present application does not require a large amount of manually labeled data, reduces the system development and maintenance costs, and quickly generates accurate SQL statements through multi-step processing and optimization algorithms, which can improve the efficiency of converting natural language queries into SQL statements.

[0060] See also Figure 1, which is a structural diagram of a query system based on natural language to SQL provided in an embodiment of the present application. The query system includes a data access layer, a preprocessing module, a deep learning model, an optimization module, a database interface module and an application layer.

[0061] The data access layer is used to receive a natural language query request from a user. It is understandable that the natural language query request includes a natural language input by the user.

[0062] In a possible implementation, the data access layer may include:

[0063] The authentication submodule is used to verify the identity of the user. Specifically, JWT (JSON WebToken) can be used for user identity authentication to ensure that only authorized users can access the system. In the specific implementation process, a library such as jsonwebtoken can be used to generate and verify tokens.

[0064] The input validation submodule is used to validate the legitimacy of the natural language input by the user using regular expressions and predefined rules. Specifically, regular expressions and predefined rules can be used to check the legitimacy of the input to prevent malicious input such as SQL injection. Middleware such as express-validator can be used to implement input validation on the server side, and basic input checks should also be performed on the client side.

[0065] The load balancing submodule is used to allocate system resources for the natural language query request using a container orchestration tool. Specifically, in a high concurrency situation, the Kubernetes container orchestration tool or the Docker Swarm container orchestration tool is used for container orchestration, which can automatically adjust the number of containers according to the system load, thereby reasonably allocating system resources.

[0066] The logging submodule is used to record the natural language query request. Specifically, the ELK (Elasticsearch, Logstash, Kibana) stack can be used to record user queries and system operations for subsequent analysis and auditing.

[0067] In addition to the above submodules, the data access layer can also include a request routing submodule, which is used to use the Nginx web server or Apache software as a reverse proxy to distribute user requests to the corresponding processing modules. Nginx's high performance and low resource consumption make it an ideal choice. You can configure Nginx's location directive to implement URL-based request routing.

[0068] The preprocessing module is used to preprocess the natural language input by the user to obtain the preprocessed natural language.

[0069] It is understandable that the preprocessing module is responsible for initially processing the input natural language query and providing a structured input for the subsequent deep learning model.

[0070] In a possible implementation, the preprocessing module may include:

[0071] A word segmentation sub-module, which is used to segment the natural language using a word segmenter to obtain the segmented natural language.

[0072] In practical applications, the jieba word segmenter can be used to segment Chinese input, or the NLTK library can be used to segment English input.

[0073] A part-of-speech tagging sub-module, which is used to perform part-of-speech tagging on the segmented natural language to obtain the tagged natural language.

[0074] Specifically, the Stanford CoreNLP toolkit or the spaCy library can be used for part-of-speech tagging to identify the part of speech of each word, which helps subsequent entity recognition and semantic analysis.

[0075] An entity recognition sub-module, which is used to perform entity recognition on the tagged natural language to obtain the recognized natural language. Specifically, named entity recognition (NER) technology is used to identify key entities in the query, such as table names, column names, conditional values, etc. A pre-trained BERT model can be used for fine-tuning to achieve high-accuracy entity recognition.

[0076] A stop word removal sub-module, which is used to remove stop words from the recognized natural language to obtain the natural language after stop word removal.

[0077] Specifically, the stop word removal sub-module can use a predefined stop word list to remove common words that are not important for SQL generation, such as "的" (de) and "是" (shi), etc., to reduce noise. It should be noted that the stop word list can be customized according to specific application scenarios.

[0078] A conversion sub-module, which is used to convert the natural language after stop word removal into word vectors using a pre-trained word vector model as the input for the deep learning model.

[0079] Specifically, the conversion sub-module can use a pre-trained word vector model (such as Word2Vec, GloVe, or FastText) to convert words into vector representations and provide numerical inputs for the deep learning model. It is understandable that word vectors can capture the semantic relationships between words and help the model understand the query intent.

[0080] In addition to the above sub-modules, the preprocessing module may further include:

[0081] The special symbol processing submodule is used to process special symbols and numbers in queries using regular expressions to ensure that they are correctly identified and processed. This is especially important for processing queries containing special formats such as dates and currencies.

[0082] The spell checking and correction submodule is used to use the Hunspell library or the Symspell algorithm to spell check the input, correct possible spelling errors, and improve the robustness of the system. It can be understood that the Symspell algorithm has a higher efficiency when processing large-scale text.

[0083] The context matching submodule is used to determine the context information of the query based on the configured context matching rules using a rule-based method or a machine learning model (such as a decision tree) to provide context for subsequent processing. It can be understood that by determining the context information of the query, it helps to resolve ambiguity and improve the accuracy of query understanding.

[0084] A deep learning model is used to convert preprocessed natural language into SQL statements, wherein the deep learning model includes an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result.

[0085] It should be noted that the deep learning model is responsible for understanding the semantics of natural language queries and providing the basis for SQL generation.

[0086] The deep learning model may be a BERT (Bidirectional Encoder Representations from Transformers) model.

[0087] The BERT model can consider both left and right contexts to better capture the semantics of the query. This is particularly important for understanding complex queries, because the meaning of certain words in the query may depend on the context of other parts. Moreover, the BERT model can be pre-trained on a large-scale general corpus and then fine-tuned on a specific NL2SQL task to improve model performance. This paradigm allows the model to learn general language representations using large-scale unlabeled data and then quickly adapt to specific tasks on a small amount of labeled data.

[0088] The architecture of the deep learning model may include:

[0089] Input layer: takes the preprocessed query token sequence and database schema information as input. Use a special separator [SEP] to distinguish different types of input.

[0090] BERT encoding layer: A 12-layer Transformer encoder is used to process the input. Each Transformer layer contains a multi-head self-attention mechanism and a feed-forward neural network.

[0091] Task-specific layers: Add NL2SQL task-specific layers on top of the BERT encoder, such as sequence labeling layers, classification layers, etc. These layers are responsible for converting BERT’s general representations into task-specific outputs.

[0092] Output layer: Generates the components of SQL queries, such as SELECT clause, WHERE clause, etc. Pointer networks or classifiers can be used to generate these components.

[0093] In a possible implementation, the deep learning model is a pre-trained language model based on Transformer, and the training process of the deep learning model includes:

[0094] Acquire a training data set, wherein the training data set includes a text corpus;

[0095] The deep learning model is trained using the training data set, and the model parameters of the deep learning model are optimized based on the PyTorch framework or the TensorFlow framework until the model convergence condition is reached, thereby obtaining the deep learning model.

[0096] It should be noted that the model convergence condition can be set according to actual conditions. For example, when the change in model parameters between two iterations is less than or equal to a change threshold, it is considered that the model convergence condition is met.

[0097] In the specific training process, you can use the Hugging Face Transformers library to load the pre-trained BERT model, use the PyTorch framework or TensorFlow framework to fine-tune and train the model, and use NVIDIA's Apex library to improve training efficiency. Furthermore, you can use gradient accumulation and gradient clipping techniques to process long sequence inputs. Use the AdamW optimizer and learning rate warm-up strategy for model training.

[0098] In order to further generate more accurate SQL statements, in a possible implementation, the query system may also include an SQL generation module for converting the output of the deep learning model into actual SQL query statements. This module uses a tree-based model approach and can better handle complex query structures.

[0099] SQL generation module can specifically include:

[0100] The semantic parser submodule is used to convert the model output into a structured semantic representation using a neural network based on the attention mechanism. In the specific implementation process, the SQL generation module can use PyTorch's nn.MultiheadAttention module, which can capture the dependencies between different parts of the query and help generate more accurate SQL structures.

[0101] The SQL template generator submodule is used to select the appropriate SQL template based on semantic representation and implement the template selection logic using decision tree or random forest algorithm. It can quickly select the most suitable template according to the characteristics of the query (such as the entity type and keywords included).

[0102] The slot filler submodule is used to fill the recognized entities into the corresponding positions of the SQL template using the conditional random field (CRF) model. It can be understood that the CRF model can consider the dependencies between tags and is suitable for processing sequence labeling tasks.

[0103] The nested query processor submodule is used to process complex nested SQL query structures using a recursive neural network (RNN). The nested query processor submodule can be implemented using LSTM or GRU units. These models are able to handle long-term dependencies and are suitable for generating SQL queries with multi-layer nested structures.

[0104] The aggregate function processor submodule is used to identify and process aggregate functions such as COUNT, SUM, AVG, etc. using a rule-based approach. It can quickly and accurately process common aggregate operations without the need for complex machine learning models.

[0105] The condition generator submodule is used to generate conditional clauses such as WHERE and HAVING using a method based on a graph neural network (GNN). The condition generator submodule can be implemented using the PyTorch Geometric library. GNN can effectively handle the relationship between tables and generate complex conditional clauses.

[0106] The join processor submodule is used to process table join operations during multi-table queries using an attention-based model. It can learn the relationship between different tables and automatically generate appropriate JOIN clauses.

[0107] Sorting and grouping processor submodules for processing ORDER BY and GROUP BY clauses using a sequence-to-sequence (Seq2Seq) model.

[0108] It can be understood that the Seq2Seq model can flexibly generate outputs of different lengths and is suitable for processing such variable-length clauses.

[0109] The limit processor submodule is used to process query limits such as LIMIT and OFFSET using a rule-based approach. These operations usually have clear syntax rules, and using a rule-based approach is more efficient and accurate.

[0110] After the deep learning model converts the preprocessed natural language into an SQL statement, the query system further includes an optimization module for optimizing the SQL statement to obtain an optimized SQL statement.

[0111] It can be understood that the optimization module is used to optimize and verify the generated SQL statements to ensure the correctness and efficiency of the queries.

[0112] Specifically, in a possible implementation, the optimization module includes:

[0113] The grammar checking submodule is used to check the grammar of the SQL statement.

[0114] Specifically, the syntax checking submodule can use a SQL parser library (such as SQLParser) to verify the correctness of the generated SQL syntax, capture and correct syntax errors that may occur during the generation process, and ensure that the generated SQL can be correctly executed by the database system.

[0115] The semantic verification submodule is used to use a machine learning model to verify whether the SQL statement conforms to the user's intention.

[0116] Specifically, the semantic verification submodule can use a rule-based system combined with a machine learning model to ensure that the generated SQL semantically conforms to the user's intention. This can include checking whether the query results are consistent with the user's expectations and whether there are logical contradictions. For example, a pre-trained natural language inference (NLI) model can be used to compare the semantic consistency of the original query and the generated SQL. At the same time, a domain-specific rule library is established to capture common semantic errors and inconsistencies.

[0117] The query optimization submodule is used to optimize the query performance of the SQL statement using an optimizer.

[0118] Specifically, the query optimization submodule can use a cost-based optimizer to analyze the execution plan and optimize query performance, which may include rewriting the query structure, adjusting the connection order, adding appropriate index hints, etc.

[0119] The query execution plan generation submodule is used to generate a query execution plan for the SQL statement and estimate the query cost.

[0120] In addition to the above submodules, the optimization module can also include:

[0121] The security check submodule is used to verify input using precompiled statements and bind variables, which can effectively prevent SQL injection attacks.

[0122] The query simplification submodule is used to simplify complex queries using logic optimization techniques and remove redundant subqueries and conditions. Specifically, this can be achieved by using equivalent rewriting rules, such as merging identical subqueries, eliminating conditions that are always true or always false, etc. This can be achieved by using a tree-structured query representation and applying a series of transformation rules.

[0123] The index suggestion submodule is used to analyze query patterns and recommend indexes for frequently accessed columns. For example, machine learning algorithms such as decision trees or random forests can be used to predict which columns are suitable for index creation based on historical query patterns and table structure characteristics. This process is dynamic and updates index suggestions as query patterns change.

[0124] The query result estimation submodule is used to estimate the size and execution time of query results using statistical methods or sampling techniques, which can help users understand the possible resource consumption before executing large queries. In specific implementations, histograms or sketch algorithms (such as HyperLogLog) can be used to estimate cardinality, and machine learning models (such as regression trees) can be used to predict query execution time.

[0125] The parallelization suggestion submodule is used to provide parallelization strategy suggestions for queries suitable for parallel execution. This can include identifying subqueries that can be executed in parallel and recommending appropriate parallelism. In specific implementation, a rule-based system can be used to identify independent query parts and determine the best parallel strategy in combination with analysis of data distribution and system resources.

[0126] The query log analysis submodule is used to collect and analyze query execution logs to identify performance bottlenecks and optimization opportunities. In specific implementations, time series analysis techniques and anomaly detection algorithms can be used to identify abnormal query patterns and performance degradation. This information can be used to continuously improve query optimization strategies and system configurations.

[0127] The dynamic plan selection submodule is used to implement a dynamic query plan selection mechanism to select the best execution plan based on runtime conditions. This can be done by collecting query execution statistics and using an online learning algorithm to select the best plan at runtime. This approach can adapt to changes in data distribution and system load, providing more stable query performance.

[0128] After obtaining the optimized SQL statement through the optimization module, the system further includes a database interface module for executing the optimized SQL statement and returning the query result corresponding to the optimized SQL statement.

[0129] It can be understood that the database interface module is a bridge connecting the query system and the actual database system, and is used to execute the generated SQL query and return the result.

[0130] In a possible implementation, the database interface module may include:

[0131] The connection pool management submodule is used to dynamically adjust the number of connections in the connection pool between the query system and the database according to the concurrency situation.

[0132] Specifically, the connection pool management submodule uses efficient connection pool technology, such as HikariCP or Apache DBCP, to manage database connections. Connection pools can reduce the overhead of frequently establishing and closing connections and improve system response speed. When implementing, you should consider dynamically adjusting the size of the connection pool to adapt to different load conditions. You can use monitoring tools to collect connection usage statistics, combined with machine learning algorithms (such as time series prediction) to predict connection needs and dynamically adjust the pool size.

[0133] The query execution submodule is used to execute SQL queries using asynchronous execution mode.

[0134] Specifically, the query execution submodule uses asynchronous execution mode to execute SQL queries to avoid long queries blocking the system. You can use Java's CompletableFuture library or Scala's Future library to implement asynchronous operations. For complex queries, consider implementing segmented query execution, splitting large queries into multiple small queries for parallel execution, and then merging the results.

[0135] The cache submodule is used to cache query plans and result sets.

[0136] Specifically, the cache submodule uses distributed cache solutions such as Redis or Memcached, combining the locality principle and LRU (least recently used) strategy to manage cache content. For frequently changing data, a time- and event-based cache invalidation mechanism is implemented.

[0137] In addition to the above sub-modules, the database interface module may also include:

[0138] The result set processing submodule is used to implement an efficient result set processing mechanism, including streaming large result sets and result set paging. For large result sets, you can use cursors or streaming APIs to avoid loading all data into memory at once. Result set paging can use the "scroll cursor" mode to obtain the next page of data only when the user requests it.

[0139] The transaction management submodule is used to implement a transaction management system that supports distributed transactions and ensure query consistency across multiple data sources. Distributed transactions can be implemented using the two-phase commit (2PC) protocol or more efficient protocols such as Paxos or Raft. For long transactions, a timeout mechanism and automatic rollback strategy are implemented to prevent resources from being locked for a long time.

[0140] The security management submodule is used to implement fine-grained access control and data desensitization mechanisms.

[0141] In the specific implementation, the security management submodule uses the role-based access control (RBAC) or attribute-based access control (ABAC) model to manage user permissions. Dynamic desensitization is implemented for sensitive data, and real-time desensitization is performed before the query results are returned according to the user's permission level.

[0142] The multi-data source management submodule is used to support various types of database systems, including relational databases and NoSQL databases. A unified abstraction layer is implemented, such as Java's JDBC or a higher-level ORM framework such as MyBatis, to provide a consistent interface. For queries across data sources, a federated query engine is implemented to decompose and distribute complex queries to different data sources for execution.

[0143] The performance monitoring submodule is used to implement real-time performance monitoring and alarm systems, collecting indicators such as query execution time and resource usage.

[0144] In the specific implementation process, the performance monitoring submodule can use open source monitoring tools such as Prometheus combined with Grafana for data collection and visualization, thereby implementing a machine learning-based anomaly detection algorithm to automatically identify performance degradation and potential problems.

[0145] The metadata management submodule is used to maintain a cache of metadata information such as database schema, table structure, index, etc. It implements a real-time synchronization mechanism for metadata changes to ensure that the schema information used by the NL2SQL system is always up to date. The metadata management submodule can use a publish-subscribe model to trigger updates when the database schema changes.

[0146] The query rewrite submodule is used to implement the query rewrite engine, which can be optimized according to database characteristics and query patterns. This can include converting complex subqueries into join operations, using materialized views to speed up queries, etc. The query rewrite submodule can use a rule-based system combined with a cost model to make query rewrite decisions.

[0147] After the query result corresponding to the optimized SQL statement is returned through the database interface module, the query system also includes an application layer for displaying the query result.

[0148] In a possible implementation, the application layer includes:

[0149] The query visualization submodule is used to provide real-time preview function.

[0150] Specifically, the query visualization submodule implements a graphical query builder, allowing users to build complex queries through drag-and-drop operations. In the specific implementation process, the query visualization submodule can use D3.js or ECharts libraries to create an interactive graphical interface to display the database model and query structure, and provide a real-time preview function, allowing users to instantly see the generated SQL statements and estimated result sets during the query construction process.

[0151] The result display submodule is used to display the query results.

[0152] Specifically, the result display submodule implements flexible data visualization functions, supports multiple chart types and interactive data exploration. It integrates data visualization libraries such as Plotly.js or Highcharts, provides rich chart options, and thus implements dynamic data loading and lazy loading technology to optimize the display performance of large data sets.

[0153] The permission management submodule is used to manage user access rights to different data sources and functions.

[0154] Specifically, the permission management submodule can use JWT (JSON Web Token) for identity authentication and manage fine-grained permissions in combination with the RBAC model. It implements the single sign-on (SSO) function to improve user experience and security.

[0155] The feedback submodule is used to collect users' evaluation and improvement suggestions on the generated query results.

[0156] Specifically, the feedback submodule can use a machine learning model to analyze user feedback and continuously improve the accuracy of NL2SQL conversion.

[0157] In addition to the above submodules, the application layer can also include:

[0158] The user interface submodule is used to design intuitive and responsive web interfaces that support access from multiple devices. In specific implementations, the user interface submodule can use modern front-end frameworks such as React or Vue.js to build single-page applications (SPAs) to provide a smooth user experience, consider implementing progressive web application (PWA) technology, and provide offline access and push notifications.

[0159] The smart suggestion submodule is used to provide real-time query suggestions and auto-completion functions based on the user's historical queries and current input.

[0160] In specific implementations, the smart suggestion submodule can use search engine technologies such as Elasticsearch or Algolia to implement a fast-response suggestion system, combined with deep learning models such as BERT to understand user intent and provide more accurate suggestions.

[0161] The collaboration submodule is used to support multi-user collaboration, including query sharing, result discussion and other functions, and implements a real-time collaboration system based on WebSocket, allowing users to share and edit queries in real time. It integrates version control functions, records the modification history of queries, and supports rollback and comparison of different versions.

[0162] The personalized settings submodule is used to set user preferences, including interface themes, default data sources, result display formats, etc. It uses browser local storage (such as localStorage) to save user settings and combines server-side user configuration to achieve a consistent experience across devices.

[0163] The API interface submodule is used to provide a RESTful API, allowing third-party applications to integrate NL2SQL functions. The API is documented using the OpenAPI (Swagger) specification to facilitate developers to understand and use it.

[0164] The multi-language support submodule is used to implement internationalization (i18n) support, allowing the interface and prompt information to be displayed in multiple languages. In the specific implementation, the multi-language support submodule can use libraries such as react-intl to manage translation resources, support dynamic language switching, and automatic language selection based on the user's geographic location.

[0165] The error handling submodule is used to implement a comprehensive error handling mechanism and a detailed logging system, using a centralized log management system such as the ELK stack (Elasticsearch, Logstash, Kibana) to collect and analyze logs. It implements user-friendly error prompts and provides a detailed error reporting function to facilitate developers to quickly locate and solve problems.

[0166] The performance optimization submodule is used to implement front-end performance optimization measures, such as code segmentation, lazy loading, resource compression, etc. In the specific implementation, the performance optimization submodule uses Webpack for module packaging and optimization, uses server-side rendering (SSR) technology to increase the first screen loading speed, implements browser caching strategy, and reduces unnecessary network requests.

[0167] It can be seen from the above technical solution that the query system provided by the embodiment of the present application has the following advantages:

[0168] 1. User-friendliness: Allows users to use natural language to query without having to master complex SQL syntax.

[0169] 2. Efficiency: Through multi-step processing and optimization algorithms, accurate SQL statements can be generated quickly.

[0170] 3. Flexibility: Able to handle various complex queries, including multi-table joins, nested queries, etc.

[0171] 4. Scalability: Able to adapt to different database environments and business needs.

[0172] 5. Low cost: It does not require a large amount of manual annotation data, which reduces the cost of system development and maintenance.

[0173] 6. Intelligence: Able to understand ambiguous expressions and improve query accuracy through context analysis.

[0174] See also Figure 2 , which is a flow chart of a natural language to SQL query method provided in an embodiment of the present application. The method is applied to the natural language to SQL query system described in the above embodiment.

[0175] The method may include:

[0176] Step S201: receiving a natural language query request, wherein the request includes a natural language input by a user.

[0177] In this step, the query system receives a natural language query request, wherein the request includes natural language input by the user.

[0178] Step S202: preprocessing the natural language to obtain preprocessed natural language.

[0179] After receiving the natural language input by the user, the query system preprocesses the natural language to obtain the preprocessed natural language.

[0180] It is understandable that by preprocessing natural language, structured input is provided for subsequent deep learning models.

[0181] Step S203: Use a deep learning model to convert the preprocessed natural language into an SQL statement, the deep learning model includes an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result.

[0182] In this embodiment, after the preprocessed natural language is obtained through step S202, the preprocessed natural language is input into the deep learning model, and the preprocessed natural language is converted into SQL statements using the deep learning model.

[0183] Among them, the deep learning model includes an encoder-decoder structure, in which the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result.

[0184] Step S204: Optimize the SQL statement to obtain an optimized SQL statement.

[0185] In this step, after the preprocessed natural language is converted into an SQL statement in step S203, the SQL statement is further optimized and verified to ensure the correctness and efficiency of the SQL query.

[0186] Step S205: Execute the optimized SQL statement and return the query result corresponding to the optimized SQL statement.

[0187] Step S206: display the query result.

[0188] It can be seen from the above technical solution that the embodiment of the present application does not require a large amount of manually labeled data, which reduces the system development and maintenance costs. Through multi-step processing and optimization algorithms, accurate SQL statements can be quickly generated, which can improve the efficiency of converting natural language queries into SQL statements.

[0189] An embodiment of the present application provides an electronic device, characterized in that it includes:

[0190] A memory for storing one or more programs;

[0191] The processor implements the natural language to SQL query method as described in the above embodiment when the one or more programs are executed by the processor.

[0192] An embodiment of the present application provides a computer storage medium, in which a program is stored. When the program is executed by a processor, the natural language to SQL query method described in the above embodiment is implemented.

[0193] The "first" and "second" in the names such as "first" and "second" (if any) mentioned in the embodiments of the present application are only used as name identifiers and do not represent the first or second in order.

[0194] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0195] Those skilled in the art can understand that the flowchart shown in the figure is only an example in which the embodiments of the present application can be implemented, and the scope of application of the embodiments of the present application is not limited by any aspect of the flowchart.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

[0197] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0198] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A query system based on natural language to SQL, characterized in that: The system comprises: Data access layer, used to receive natural language query requests from users; A preprocessing module, used to preprocess the natural language input by the user to obtain the preprocessed natural language; A deep learning model, used to convert the preprocessed natural language into SQL statements, the deep learning model comprising an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result; An optimization module, used to optimize the SQL statement to obtain an optimized SQL statement; A database interface module, used to execute the optimized SQL statement and return the query result corresponding to the optimized SQL statement; The application layer is used to display the query results.

2. The system according to claim 1, characterized in that The data access layer includes: The authentication submodule is used to verify the identity of the user; The input validation submodule is used to validate the legitimacy of the natural language input by the user using regular expressions and predefined rules; A load balancing submodule, configured to allocate system resources to the natural language query request using a container orchestration tool; The logging submodule is used to log the natural language query request.

3. The system according to claim 1, characterized in that The preprocessing module comprises: A word segmentation submodule, used for using a word segmenter to segment the natural language to obtain the natural language after word segmentation; A part-of-speech tagging submodule is used to perform part-of-speech tagging on the natural language after word segmentation to obtain the tagged natural language; An entity recognition submodule, used to perform entity recognition on the annotated natural language to obtain recognized natural language; A stop word removal submodule, used to remove stop words from the recognized natural language to obtain the natural language after the stop words are removed; The conversion submodule is used to use a pre-trained word vector model to convert the natural language after the stop words are removed into a word vector as an input of the deep learning model.

4. The system according to claim 1, characterized in that The deep learning model is a pre-trained language model based on Transformer. The training process of the deep learning model includes: Acquire a training data set, wherein the training data set includes a text corpus; The deep learning model is trained using the training data set, and the model parameters of the deep learning model are optimized based on the PyTorch framework or the TensorFlow framework until the model convergence condition is reached, thereby obtaining the deep learning model.

5. The system according to claim 1, characterized in that The optimization module includes: A grammar checking submodule, used to check the grammar of the SQL statement; A semantic verification submodule is used to verify whether the SQL statement conforms to the user's intention by using a machine learning model; A query optimization submodule, used to optimize the query performance of the SQL statement using an optimizer; The query execution plan generation submodule is used to generate a query execution plan for the SQL statement and estimate the query cost.

6. The system according to claim 1, characterized in that The database interface module includes: A connection pool management submodule, used to dynamically adjust the number of connections in the connection pool between the query system and the database according to the concurrency situation; The query execution submodule is used to execute SQL queries using asynchronous execution mode; The cache submodule is used to cache query plans and result sets.

7. The system according to claim 1, characterized in that The application layer includes: Query visualization submodule, used to provide real-time preview function; A result display submodule, used to display the query results; The permission management submodule is used to manage user access rights to different data sources and functions; The feedback submodule is used to collect users' evaluation and improvement suggestions on the generated query results.

8. A natural language to SQL query method, characterized in that: Applied to the system as claimed in claims 1-7, the method comprises: receiving a natural language query request, wherein the request includes natural language input by a user; Preprocessing the natural language to obtain preprocessed natural language; The preprocessed natural language is converted into an SQL statement using a deep learning model, wherein the deep learning model includes an encoder-decoder structure, wherein the encoder is used to encode the input natural language to obtain a corresponding encoding result, and the decoder is used to generate an SQL statement based on the encoding result; Optimizing the SQL statement to obtain an optimized SQL statement; Execute the optimized SQL statement and return the query result corresponding to the optimized SQL statement; The query result is displayed.

9. An electronic device, characterized in that: include: A memory for storing one or more programs; The processor implements the query method according to claim 8 when the one or more programs are executed by the processor.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the query method according to claim 8 is implemented.

Images