System and method for enhanced natural language to SQL generation in multi-table question answering

Abstract

The embodiment provides a method for executing a query using NL2SQL. The method includes: receiving a natural language question associated with information stored in at least one table of a database; generating a prompt based on the natural language question, the prompt including instructions for causing a large language model (LLM) to perform a keyword extraction and task decomposition process; performing the keyword extraction and task decomposition process via the LLM using the instructions to generate a set of keywords and task decomposition data; retrieving a set of entities from the at least one table based on the set of keywords; generating a second prompt based on the set of entities, the task decomposition data, and the question; generating a query statement via the LLM based on the second prompt; and executing the query statement against the at least one table to obtain the answer to the question from the at least one table.

Description

Technical Field

[0001] The present disclosure relates to form question answering (form QA) systems and methods, and more particularly, to improved methods and computing architectures for performing form QA. Background Art

[0002] Table Question Answering (Table QA) is a task that helps users who are not proficient in coding skills or advanced spreadsheet software to retrieve complex table data through question answering. One method to perform Table QA is Natural Language to SQL (NL2SQL), which is a process of translating natural language queries from users into Structured Query Language (SQL) commands that can be used to extract data from one or more data sources such as databases or tables. NL2SQL allows users to interact with a database based on questions written in everyday language (i.e., natural language) without requiring users to have SQL knowledge.

[0003] Recent studies have shown that NL2SQL approaches that leverage large language models (LLMs) significantly outperform other rule-based or neural network-based approaches. A direct approach may involve submitting prompts to an LLM (e.g., OpenAI's GPT-4o) to perform the relevant task. However, this approach often results in logical errors in SQL statements and inaccurate field identification. This approach also has difficulties managing multi-table relationships. It has been hypothesized that these problems are caused by the LLM's insufficient understanding of user questions in table QA scenarios.

[0004] Previous work in this area (such as TAT-QA by Zhu et al., FinQA by Chen et al., and MVGE by Ma et al.) has adopted internal context learning (ICL), fine-tuning, and pre-training methods. Although these efforts have achieved significant progress, these methods have problems in adaptability in multi-table scenarios. Recently, methods have been developed to convert tabular data into graph structures to enhance reasoning. Although promising, these methods are still time-consuming, resource-intensive, and face challenges in accurate graph construction.

[0005] At the same time, NL2SQL research, which translates natural language questions into SQL queries, provides a more efficient and cost-effective solution compared to the above methods. NL2SQL techniques are mainly divided into rule-based methods, neural network-based methods, pre-trained language model (PLM)-based methods, and large language model (LLM)-based methods. Initially, rule-based methods prevailed, utilizing predefined rules or semantic parsers, but these methods were quickly replaced by more scalable neural network techniques. By 2017, PLM methods, especially those using models such as BERT, took precedence. Currently, LLMs represented by GPT-4 dominate the field, powering advanced methods such as CHESS by Talaei et al., DAIL-SQL by Gao et al., and MAC-SQL by Wang et al. These advanced methods are characterized by specialized modules such as filters, evaluators, and self-correction mechanisms to refine their outputs. Despite the complexity of LLM-based methods, they still face challenges such as low accuracy, high operating costs, and considerable running time, which limits their practical application.

[0006] Natural language understanding (NLU) is the cornerstone of AI, enabling machines to interpret and process human language. This field includes a wide range of tasks from keyword extraction to complex question answering. The emergence of LLMs such as Gemini-Pro, GPT-4, and Mistral has completely revolutionized NLU and pushed the boundaries of machine understanding.

[0007] To further enhance NLU capabilities, researchers have studied various innovative approaches. These approaches include complex text alignment, integration of human-written explanations, and advanced reasoning techniques such as thought chains (COTs), thought trees, and thought buffers. Specialized datasets such as Adversarial NLI and SemEval-2024 Task 2 have been created to evaluate and refine the NLU proficiency of LLMs.

[0008] Despite these advances, significant challenges remain for NLU, especially in the area of ​​table QA. While LLMs demonstrate impressive reasoning capabilities, they often struggle to extract precise information and reason from tabular data, such as data stored in one or more tables in a database. A key limitation is that they cannot distinguish between meaningful and meaningless language in user queries, and cannot consistently identify and extract relevant keywords corresponding to filters, column names, or table names in a database. This shortcoming highlights the urgent need for specialized datasets and fine-tuned models tailored specifically for NLU in table QA. Summary of the invention

[0009] New systems, methods, and computer-readable storage media that support functionality for executing queries using NL2SQL are described. In one aspect, a system for executing queries using NL2SQL is disclosed. The system includes: a memory; and one or more processors, the one or more processors being communicatively coupled to the memory. The one or more processors may be configured to receive a natural language question associated with information stored in at least one table of a database, and to generate a prompt based on the natural language question. The prompt may include a first instruction corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question, and a second instruction corresponding to a task decomposition process configured to determine one or more tasks for answering the question. In one aspect, the second instruction includes one or more examples, and each of the one or more examples includes a natural language question, a main task derived from the natural language question, and one or more subtasks for implementing the main task derived from the natural language question. The one or more processors may be configured to perform a keyword extraction process and a task decomposition process via an LLM based on the first instruction and the second instruction of the prompt to generate a set of keywords and task decomposition data. In one aspect, the set of keywords includes object keywords and implementation keywords. In one aspect, the task decomposition process is configured to determine a main task and one or more subtasks for implementing the main task.

[0010] The one or more processors may be configured to retrieve a set of entities from the at least one table based on the set of keywords, and generate a second prompt based on the set of entities, the task decomposition data, and the question. In one aspect, the one or more processors are configured to: generate a set of keyword embeddings based on the set of keywords and generate a set of table embeddings based on the data pattern, and determine a set of initial entities based at least in part on the set of keyword embeddings and the set of table embeddings. In one aspect, the set of entities is determined at least in part based on the ranking of a set of initial entities. The second prompt may include a third instruction for generating a query statement to extract the answer to the question from the at least one table. In one aspect, the third instruction of the second prompt includes information associated with the set of entities, the data pattern of the at least one table of the database, the question, and the task decomposition data. The one or more processors may be configured to generate the query statement via the LLM based on the second prompt, and execute the query statement for the at least one table to obtain the answer to the question from the at least one table.

[0011] In one aspect, the LLM may be fine-tuned based on a data set including a plurality of question and query statement pairs. Each question and query statement pair may include a sample natural language question and a benchmark truth query statement corresponding to the sample natural language question. In one aspect, the one or more processors are configured to: receive a query result in response to executing the query statement; and output the query result. In one aspect, the query result may include an answer to the question from the at least one table. The one or more processors are configured to: determine whether the query result includes an error, and execute a correction process configured to generate a corrected query statement. The corrected query statement may be configured to resolve at least a portion of the error. In one aspect, the correction process is iteratively performed until a stopping criterion is reached.

[0012] A method for executing a query using NL2SQL includes: receiving, by one or more processors, a natural language question associated with information stored in at least one table of a database; and generating, by the one or more processors, a prompt based on the natural language question. In one aspect, the prompt includes: a first instruction corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question, and a second instruction corresponding to a task decomposition process configured to determine one or more tasks for answering the question. The prompt may include a first instruction corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question, and a second instruction corresponding to a task decomposition process configured to determine one or more tasks for answering the question. In one aspect, the second instruction includes one or more examples, and each of the one or more examples includes a natural language question, a main task derived from the natural language question, and one or more subtasks for implementing the main task derived from the natural language question. The method may include executing a keyword extraction process and a task decomposition process via an LLM based on the first instruction and the second instruction of the prompt to generate a set of keywords and task decomposition data. In one aspect, the set of keywords includes object keywords and implementation keywords. In one aspect, the task decomposition process is configured to determine a main task and one or more subtasks for implementing the main task.

[0013] The method may include retrieving a set of entities from the at least one table based on the set of keywords, and generating a second prompt based on the set of entities, the task decomposition data, and the question. In one aspect, the method may include: generating a set of keyword embeddings based on the set of keywords and generating a set of table embeddings based on the data schema, and determining a set of initial entities based at least in part on the set of keyword embeddings and the set of table embeddings. In one aspect, the set of entities is determined based at least in part on the ranking of a set of initial entities. The second prompt may include third instructions for generating a query statement to extract an answer to the question from the at least one table. In one aspect, the third instructions of the second prompt include information associated with the set of entities, the data schema of the at least one table of the database, the question, and the task decomposition data. The method may include generating the query statement via the LLM based on the second prompt, and executing the query statement against the at least one table to obtain an answer to the question from the at least one table.

[0014] In one aspect, the LLM may be fine-tuned based on a data set including a plurality of question and query statement pairs. Each question and query statement pair may include a sample natural language question and a baseline truth query statement corresponding to the sample natural language question. In one aspect, the method may include: receiving a query result in response to executing the query statement; and outputting the query result. The query result may include an answer to the question from the at least one table. The method may include: determining whether the query result includes an error, and executing a correction process configured to generate a corrected query statement. The corrected query statement may be configured to resolve at least a portion of the error. In one aspect, the correction process is iteratively performed until a stopping criterion is reached.

[0015] A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations to execute a query using NL2SQL. In one aspect, the operations include: receiving a natural language question associated with information stored in at least one table of a database, and generating a prompt based on the natural language question. In one aspect, the prompt includes: a first instruction corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question, and a second instruction corresponding to a task decomposition process configured to determine one or more tasks for answering the question. The prompt may include a first instruction corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question, and a second instruction corresponding to a task decomposition process configured to determine one or more tasks for answering the question. In one aspect, the second instruction includes one or more examples, and each of the one or more examples includes a natural language question, a main task derived from the natural language question, and one or more subtasks for implementing the main task derived from the natural language question. The operations may include executing a keyword extraction process and a task decomposition process via an LLM based on the first instruction and the second instruction of the prompt to generate a set of keywords and task decomposition data. In one aspect, the set of keywords includes object keywords and implementation keywords. In one aspect, the task decomposition process is configured to determine a main task and one or more subtasks for implementing the main task.

[0016] The operations may include retrieving a set of entities from the at least one table based on the set of keywords, and generating a second prompt based on the set of entities, the task decomposition data, and the question. In one aspect, the operations include generating a set of keyword embeddings based on the set of keywords and generating a set of table embeddings based on the data schema, and determining an initial set of entities based at least in part on the set of keyword embeddings and the set of table embeddings. In one aspect, the set of entities is determined based at least in part on a ranking of a set of initial entities. The second prompt may include third instructions for generating a query statement to extract an answer to the question from the at least one table. In one aspect, the third instructions of the second prompt include information associated with the set of entities, the data schema of the at least one table of the database, the question, and the task decomposition data. The operations may include generating the query statement via the LLM based on the second prompt, and executing the query statement against the at least one table to obtain an answer to the question from the at least one table.

[0017] In one aspect, the LLM may be fine-tuned based on a data set including a plurality of question and query statement pairs. Each question and query statement pair may include a sample natural language question and a benchmark truth query statement corresponding to the sample natural language question. In one aspect, the operations include: receiving a query result in response to executing the query statement; and outputting the query result. The query result may include an answer to the question from the at least one table. The operations include: determining whether the query result includes an error, and executing a correction process configured to generate a corrected query statement. The corrected query statement may be configured to resolve at least a portion of the error. In one aspect, the correction process is iteratively performed until a stopping criterion is reached.

[0018] The foregoing has been fairly extensively summarized the features and technical advantages of the present invention so that the following detailed description of the present invention may be better understood. Additional features and advantages of the present invention will be described below, which form the subject matter of the claims of the present invention. It will be appreciated by those skilled in the art that the disclosed concepts and specific embodiments may be easily used as the basis for modifying or designing other structures for achieving the same purpose of the present invention. It will also be appreciated by those skilled in the art that such equivalent constructions do not depart from the spirit and scope of the present invention as set forth in the appended claims herein. When considered in conjunction with the accompanying drawings, the novel features that are considered to be the characteristics of the present invention with respect to the organization and method of operation of the present invention will be better understood from the following description together with further objects and advantages. However, it should be clearly understood that each of the drawings in the accompanying drawings is provided for the purpose of illustration and description only, and is not intended to be a definition of limitations of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] For a more complete understanding of the disclosed methods and apparatus, reference should be made to the embodiments illustrated in more detail in the accompanying drawings, in which:

[0020] Figure 1 is a block diagram of a system supporting functionality for executing NL2SQL according to aspects of the present disclosure;

[0021] Figure 2 is a flowchart illustrating an example process for executing NL2SQL according to aspects of the present disclosure;

[0022] Figure 3 is a block diagram of an example computing architecture for executing NL2SQL according to aspects of the present disclosure; and

[0023] Figure 4 is a flow chart of a method for executing a query using NL2SQL technology according to aspects of the present disclosure.

[0024] It should be understood that the drawings are not necessarily drawn to scale and that the disclosed embodiments are sometimes shown diagrammatically and in partial views. In some cases, details that are not necessary for understanding the disclosed methods and apparatus or that make other details difficult to perceive may have been omitted. Of course, it should be understood that the present disclosure is not limited to the specific embodiments shown herein. DETAILED DESCRIPTION

[0025] In order to effectively generate SQL from natural language input, models that excel in natural language understanding (NLU) can be utilized. According to aspects of the present disclosure, NLU for NL2SQL can involve two different areas: 1) fine-grained task decomposition; and 2) precise keyword extraction. Fine-grained task decomposition can be used for complex multi-table reasoning, and precise keyword extraction can be used to ensure accurate recognition of table names and column names (e.g., to improve the accuracy of SQL commands generated from natural language input). However, there are still challenges in using currently available LLMs (such as GPT-4o) for task decomposition and keyword extraction, because such models may generate inadequate tasks and misidentify keywords.

[0026] As a non-limiting example, assume that the natural language input on which NL2SQL is to be executed is as follows:

[0027] “How many schools with an average SAT math score of over 560 are directly charter-funded?”

[0028] Using the two domains outlined above, and providing the above example input as a prompt to GPT-4o may result in: 1.) A task decomposition that identifies the task to be performed as "identify schools with an average math score of over 560 on the SAT test," and the following set of keywords: ["schools with average scores," "math," "560," "SAT test," "charter-funded"]. As can be appreciated, GPT-4o's prompted NLU fails to fully capture the goal of the question presented in the prompt. For example, GPT-4o fails to detect that the answer of interest is "How many schools with an average math score of over 560 are charter-funded?" Generating an SQL query based on the task decomposition and keywords obtained using GPT-4o may result in the following generated SQL query:

[0029] SELECT COUNT(*)FROM satscores T1 JOIN schools T2 ON T1.cds=T2.CDSCode WHERE T1.AvgScrMath>560AND T2.FundingType='Direct'

[0030] Compared to the above example, and as will become more apparent from the following description, the concepts disclosed herein provide improved NLU for the presented problem through better decomposition of the tasks to be performed and better detection of more relevant keywords. For example, applying the concepts disclosed herein to the above natural language problem may result in a task decomposition including the following tasks:

[0031] 1. Identify schools with an average math score of more than 560;

[0032] 2. Determine whether the schools are directly charter funded; and

[0033] 3. Calculate the number of schools that receive direct charter funding.

[0034] Additionally, keywords may be identified as: ["school", "average math score", "560", "SAT test", "direct charter funding"]

[0035] As a result of the enhanced NLU provided by the concepts described herein, a SQL query may be generated as follows:

[0036] SELECT COUNT(T2.`School Code`)FROM satscores AS T1 INNER JOIN frpm AST2 ON T1.cds=T2.CDSCode WHERE T1.AvgScrMath>560AND T2.`Charter Funding Type`='Directly funded'

[0037] As can be understood from the above examples, currently available NL2SQL techniques and available models (e.g., LLMs, etc.) have several shortcomings and challenges, such as challenges associated with performing NLU on natural language problems. Solving these challenges requires specialized training because it involves understanding and manipulating structured data in a specific context, which is different from the more general natural language tasks that LLMs may be suitable for. For example, fine-grained task decomposition involves breaking down complex queries into smaller, precise steps. These steps must then be aligned with the relational schema of one or more databases so that queries can be generated using appropriate commands and variable names to extract relevant data from the databases. Additionally, precise keyword extraction is required to accurately map natural language to specific table names and column names, requiring a deep understanding of the database and its structure. An additional challenge is the lack of quantitative evaluation metrics for evaluating the NLU performance of different LLMs within the table QA domain, which has hindered progress in this specialized field.

[0038] In addition to directly applying LLM to NL2SQL, hybrid approaches that combine LLM with various modules have also shown promise. Notable examples include CHESS, TA-SQL, and MAC-SQL. However, challenges such as slow data retrieval, incorrect SQL code generation, and high operational costs remain.

[0039] As can be appreciated from the above examples, the techniques described in more detail below enhance the NLU of a table QA system for executing NL2SQL, so that SQL commands and queries generated according to the concepts described herein can more easily obtain query results that more accurately answer questions presented in natural language. In addition to the improved accuracy of information obtained as a result of SQL queries or commands generated according to the concepts disclosed herein, the time and cost of executing NL2SQL and obtaining results can also be reduced. For example, existing NL2SQL methods such as MAC-SQL, CHESS, and TA-SQL may be efficient in terms of time or cost, but not in both. In contrast, the techniques disclosed herein can execute NL2SQL in a manner that is efficient in both time and cost relative to these existing methods for executing NL2SQL.

[0040] refer to Figure 1 A block diagram of a system supporting functionality for executing NL2SQL according to aspects of the present disclosure is shown as system 100. Figure 1 As shown, the system 100 includes a computing device 110. In one aspect, the functionality described with respect to the computing device 110 may be implemented via a cloud-based deployment (e.g., one or more processes or applications running on computing resources of a network) as shown by cloud logic 152, rather than via a server or other type of computing device. Additionally or alternatively, the functionality described with respect to the computing device 110 may be implemented via multiple computing devices (e.g., multiple computing devices 110) or across a network 150. The computing device 110 includes one or more processors 112, a memory 114, one or more dynamic resource detection engines 120, one or more automation engines 122, one or more communication interfaces 124, and one or more input / output (I / O) devices 126. Each of the one or more processors 112 may be a central processing unit (CPU), a graphics processing unit (GPU), or other computing circuitry (e.g., a microcontroller, one or more application specific integrated circuits (ASICs), etc.), and each processor 112 may have one or more processing cores.

[0041] The memory 114 may include a read-only memory (ROM) device, a random access memory (RAM) device, one or more hard disk drives (HDDs), flash memory devices, solid-state drives (SSDs), network attached storage (NAS) devices, other devices configured to store data in a persistent or non-persistent state, or a combination of different memory devices. The memory 114 may store instructions 116 that, when executed by the one or more processors 112, cause the one or more processors 112 to perform operations in conjunction with the computing device 110. Figures 1 to 4 The memory 114 may also store one or more databases 118 configured to store information used to support the operation of the system 100. For example, the one or more databases 118 may store one or more tables that may be queried using commands (e.g., SQL queries or commands) as described in more detail below to extract information in response to questions presented to the computing device 110 in natural language. In one aspect, the query may also be performed against one or more remote databases (such as a database stored in the data source 140 or the database 154).

[0042] The one or more communication interfaces 124 may be configured to communicatively couple the computing device 110 to one or more networks 150 via a wired or wireless communication link according to one or more communication protocols or standards (e.g., Ethernet protocol, Transmission Control Protocol / Internet Protocol (TCP / IP), Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol, IEEE 802.16 protocol, 3rd generation (3G) communication standard, 4th generation (4G) / Long Term Evolution (LTE) communication standard, 5th generation (5G) communication standard, etc.). For example, the one or more communication interfaces 124 may communicatively couple the computing device 110 to a remote computing device such as computing device 130. A user of computing device 130 may access the functionality of computing device 110 through network 150 to perform queries on one or more databases 118 using NL2SQL functionality provided by computing device 110. For example, a user of computing device 130 may submit a question in natural language to computing device 110, and functionality of computing device 110 may convert the natural language question into SQL commands that are executed against one or more databases 118 to obtain information in response to the question presented by the user of computing device 130.

[0043] I / O devices 126 may include one or more types of devices that enable a user to receive information from or provide information to computing device 110. For example, I / O devices 126 may include one or more display devices, a keyboard, a stylus, one or more touch screens, a mouse, a track pad, a camera, one or more speakers, a haptic feedback device, or other types of devices that enable a user to receive information from or provide information to computing device 110.

[0044] The computing device 110 also includes a query engine 120 and a modeling engine 122. The modeling engine 122 can be configured to process natural language input using the NLU technology disclosed herein to extract information from the natural language input. For example, a user can provide a question to the computing device 110 in natural language form, and the modeling engine 122 can be configured to perform task decomposition and keyword extraction based on the natural language question. The query engine 120 can be configured to generate a query based on the task information and keywords extracted by the modeling engine 122. Once generated, the query engine 120 can execute a query on one or more databases 118 to obtain information that can be provided to the user in response to the presented natural language question. Exemplary operations of the query engine 120 and the modeling engine 122 are described below with reference to Figure 2 Describe in more detail.

[0045] As briefly described above, computing device 110 may be communicatively coupled to one or more computing devices 130. Each computing device 130 may include one or more processors, memory, I / O devices, communication interfaces, etc. to facilitate interaction with computing device 110 to retrieve information from data sources (e.g., one or more databases 118, data source 140, data source 154, etc.). For example, a user of computing device 130 may access functionality of computing device 110 via a web browser, mobile application, or other type of application or service. The functionality may enable a user to submit a question in natural language form to computing device 110 and then receive a response containing information responsive to the question, where the information is retrieved from the data source using functionality provided by query engine 120 and modeling engine 122.

[0046] The functionality of computing device 110 is improved over existing systems for performing table QA. Non-limiting examples of improvements provided by computing device 110 and its functionality include: 1) building a dataset specifically designed to evaluate NLU within the NL2SQL approach (referred to herein as "LAIA-NLU"), and 2) a new proxy framework designed to enhance the performance of NL2SQL in terms of accuracy, efficiency, and cost (referred to herein as "LAIA-SQL"). An ablation study, described in more detail below, was used to validate LAIA-SQL and demonstrate the effectiveness of the LAIA-NLU dataset.

[0047] Based on the above concepts, a LAIA-NLU dataset consisting of 1,500 high-quality QA pairs focusing on task decomposition and keyword extraction was created. As an example, a LAIA-SQL dataset derived from the BIRD dataset (e.g., Li et al.) was constructed. As part of the construction of this dataset, three rounds of careful manual annotation were performed. The LAIA-SQL technology was refined based on CHESS (Talaei et al.) and optimized into three main modules via the above ablation study. These three modules include: 1) User Question Understanding (UQU), 2) Entity Retrieval, and 3) Generation. For the ablation study, the LAIA-NLUer model (which is a model fine-tuned based on GPT-4o-Mini using the LAIA-NLU dataset) was developed for the UQU module to enhance understanding. Additionally, retrieval and reranking techniques are combined in the entity retrieval module to improve accuracy, and a correction process guided by task reasoning and error feedback is introduced during code generation.

[0048] Current LLMs demonstrate limited NLU capabilities in table QA, adversely affecting the ultimate accuracy of the NL2SQL process to which the LLMs can be applied. In addition, in table QA, there is no existing dataset to evaluate these models in terms of NLU. To address this gap, a LAIA-NLU dataset can be created. As a proof of concept and as mentioned above, an example LAIA-NLU dataset was generated based on the BIRD dataset, which has a proven source and extensive research use. BIRD includes 12,751 text-SQL pairs across 95 databases, totaling 33.4GB and across 37 professional fields, and is specifically designed for evaluating and training NL2SQL methods. It integrates 80 open source relational databases from platforms such as Kaggle and Relation.vit. In order to prevent data leakage, 15 additional relational databases were created for hidden test sets.

[0049] Random selection was used to select 1,500 instances from the training data of the BIRD dataset. Each instance includes a user question and a corresponding ground truth SQL query. Initially, GPT-4o was used to perform task decomposition and keyword extraction. Task decomposition involves decomposing the user question into two components: a main task and subtasks. The main task represents the main goal derived from the user question, while one or more subtasks further refine the main task. In the keyword extraction stage, keywords are divided into two categories: objects and implementations. The object category includes terms related to table names and column names in the user question, while the implementation category involves filtering criteria. On the one hand, a dictionary can be used to perform filtering. The key can represent a filtering action, and the value can specify a condition. These elements together facilitate similarity matching within the database.

[0050] However, despite the implementation of CoT and few-shot techniques, GPT-4o's performance in deciphering user queries is not ideal. For example, GPT-4o often generates redundant or incomplete tasks and extracts incorrect keywords. Therefore, further refinement of the raw data generated is required. Annotators were used to review and correct the data generated by GPT-4o. The annotation strategy requires a three-stage cyclic process to ensure cross-validation and accuracy. Before stage 1, each annotator started with a different subset (A, B, C), exchanging and reviewing the modified subsets in subsequent stages until all data was thoroughly evaluated by all annotators. The validation process includes the following three-step process:

[0051] Evaluating Task Decomposition — Annotators first manually reviewed each question to evaluate the accuracy of the main task and subtasks generated by GPT-4o. They checked the logical consistency, removed redundant tasks, and manually added any missing related tasks.

[0052] Evaluation of keyword extraction — Keywords were classified into objects and implementations. Annotators compared the keywords generated by GPT-4o with user questions and corresponding ground truth SQL elements (such as filters, table names, and column names) to ensure accuracy. They added missing keywords and removed irrelevant keywords. An initial training with 50 data points was conducted to train annotators and evaluate precision scores to maintain quality standards.

[0053] Final Rating—After three rounds of rotational evaluation, annotators rated the revised keywords and tasks using a 5-point Likert scale, where 1 indicated “unsatisfactory” and 5 indicated “excellent.” For any case where the average score was lower than 4, the annotators collaboratively discussed and finalized the revisions.

[0054] After three rounds of review, a dataset consisting of 1,500 pairs of instructions and implementations (also referred to herein as the "LAIA-NLU dataset") was created. The dataset was divided into training, validation, and test sets in a ratio of 7:2:1 to ensure robust model training and evaluation. It should be noted that other ratios for training, validation, and testing can be used, and the example ratio of 7:2:1 is provided by way of non-limiting example. The dataset can be used to fine-tune an LLM (e.g., GPT-4o-mini or another LLM) to produce a fine-tuned LLM. The fine-tuned LLM can then be used to perform keyword extraction, task decomposition, or other tasks that LLMs can be used for according to aspects of the present disclosure.

[0055] The distribution of main tasks, subtasks, and keywords was analyzed to assess the complexity of the questions. Complexity can be inferred from the number of tasks that the model needs to handle, which tests its reasoning and integration capabilities. Additionally, higher counts of keywords indicate more complex table and column settings, increasing the likelihood of errors. In the example above, it is observed that for the main task, 68.2% of the questions involve one main task, while 24.9% include two tasks, and 6.9% require three or more tasks. The subtask distribution shows that 31.7% of the questions include one to two subtasks. At the same time, 60% involve three to four subtasks, and 8.3% contain more than five subtasks. Regarding keywords, 20.3% of the questions are linked to one or two keywords, 60.6% are linked to three or four keywords, and 19.2% are linked to five or more keywords.

[0056] As described above, current state-of-the-art methods (e.g., MAC-SQL and CHESS) have advanced the field of SQL generation, but still suffer from considerable runtimes, high operational costs, and suboptimal accuracy. To address these limitations, the query engine 120 and the modeling engine 122 provide an innovative language-adaptive intelligent agent framework designed to enhance SQL generation. The agent framework provides functionality that supports user question understanding (e.g., keyword extraction and task decomposition), entity retrieval, and query generation. In one aspect, the functionality of the query engine 120 and the modeling engine 122 may be supported by the LAIA-NLU dataset described above. For example, the functionality of the query engine 120 and the modeling engine 122 may utilize one or more LLMs that have been fine-tuned using the LAIA-NLU dataset.

[0057] To illustrate the high-level concepts described above, and to refer to Figure 2 A block diagram illustrating an example NL2SQL process according to the concepts disclosed herein is shown as process 200. Process 200 may be executed by a computing device such as Figure 1 The computing device 110 is executed with the support of the query engine 120 and the modeling engine 122.

[0058] At step 202, process 200 includes receiving a natural language question. Modeling engine 122 may be configured to receive a user question as input. Modeling engine 122 may perform one or more question understanding operations to understand one or more questions included in the input. In one aspect, at step 204, the input may be incorporated into a prompt template before providing the input to modeling engine 122. For example, the input may be a natural language question, and the prompt template may incorporate the input question and instructions or examples to form a modified prompt, and the modified prompt may be provided as input to the LLM of modeling engine 122. The LLM may generate a response based on the modified prompt. The response may include the keywords extracted at step 206 and the task decomposition information extracted at 208.

[0059] During task decomposition, at step 208, the input problem may be decomposed into manageable components, thereby addressing the inherent complexity and multi-task nature of user queries. The modeling engine 122 may employ two-level COT reasoning, which decomposes the user problem into a main task and subtasks, wherein the main task represents the main goal derived from the user problem, and the subtasks further refine the main task. This may help the NL2SQL process to efficiently generate SQL code by clarifying the hierarchical structure of the task. For example, a prompt template may indicate to the LLM that the main task corresponds to the main component following "SELECT", while the subtasks correspond to operations such as "INNER JOIN", "WHERE", and "CASE WHEN" and others. However, general models such as GPT-4o sometimes decompose tasks incorrectly or generate irrelevant tasks, indicating unstable performance. To enhance stability and reliability, supervised fine-tuning for consistent task decomposition may be used.

[0060] As described above, existing methods only involve breaking down sentences into individual keywords, which often results in irrelevant keywords. During keyword extraction, at step 206, words within the input are classified into keywords within one of two different categories (object and implementation). This can improve the accuracy of the identified keywords and improve the queries generated as a result of process 200. The object keyword category covers terms associated with table names and column names found in the user's natural language questions, while the implementation involves filtering criteria represented by a dictionary in which the keys indicate the filtering action and the values ​​represent the specified conditions. In order to improve the accuracy of keyword extraction, in-context learning (ICL) techniques can be used to provide multiple examples to the LLM. On the one hand, these examples can be included in the prompt template. However, some LLMs (such as GPT-4o) tend to generate irrelevant or excessive keywords. To address this issue, a smaller model (e.g., Mistral-7B) can be fine-tuned using the LAIA-NLU dataset, which can improve the accuracy of keyword extraction. On the one hand, the system (e.g., Figure 1 The system 100 of the embodiment of the present invention can use a single LLM to perform natural language understanding (e.g., keyword extraction and task decomposition) of the received question. In additional or alternative aspects, the system can use multiple LLMs to perform natural language understanding. For illustration, GPT-4 can be used for task decomposition, and Mistral-7b can be used for keyword extraction. Note that ICL can be utilized (e.g., via instructions included in a prompt submitted to the LLM) regardless of whether the LLM is a base LLM or an LLM that has been fine-tuned (e.g., using the LAIA-NLU dataset). Note that the fine-tuned LLM can achieve different (higher) performance in both task decomposition and keyword extraction. For example, experiments (discussed below) show that the fine-tuned GPT-4 model performs best in task decomposition, and the fine-tuned Mistral-7b model performs best in keyword extraction.

[0061] After extracting keywords and breaking down the input into tasks, at step 210, a set of embeddings may be generated. Then, at step 212, entity retrieval may be performed using the set of embeddings. Entity retrieval corresponds to the process of retrieving corresponding database entities including table names, column names, table values, and text descriptions (column and value descriptions). In one aspect, tabular data corresponding to one or more databases that can be searched using NL2SQL and the functionality of the computing device 110 may be encoded and stored in the database as a set of database embeddings. Keywords obtained during the problem understanding phase (e.g., step 206) may be encoded to obtain a set of embeddings. Then, at step 212, entity retrieval may be performed using these embeddings. During entity retrieval, the embeddings corresponding to the keywords may be compared with the embeddings of the tabular data of one or more databases. Then, a set of entities may be obtained based on the similarity between the two sets of embeddings. In one aspect, the set of entities may include a threshold number of entities (e.g., 5 entities, 10 entities, or some other number of entities) that are similar or similar to the keywords, where the similarity or degree of similarity is determined based on comparing the two sets of embeddings.

[0062] Once the entities are identified, a ranking process may be performed at step 214 to rank the entities. The ranking of the entities may indicate which entity most likely corresponds to one or more of the keywords extracted in step 206. In one aspect, the modeling engine 122 may be configured to recalculate similarity scores and reorder entities before selecting the most similar entity. For illustration, at step 212, the entity retrieval process may use a retriever to retrieve the 10 most similar entities (e.g., the 10 entities in the table or database data that are most similar to each keyword included in the set of keywords), and then the similarity measures between the keywords and the 10 similar entities may be recalculated based on the embedding model of the reranker. Based on the ranking calculated by the reranker, the top "X" (e.g., top 2) most similar entities may be retained for query generation. Note that entity retrieval may be divided into two tasks: 1) database retrieval, and 2) text description retrieval.

[0063] The purpose of database retrieval is to retrieve column names and table values ​​from a database using keywords. Column names refer to the names assigned to each column, and table values ​​are the data contained in each cell of the table, excluding the column names. For the retrieval process, given a large number of database values, two methods can be used: Min-Hash+Jaccard Score (Equations 1 and 2 below) and BM25. For example, Min-Hash can be used to encode the set of keywords, and the min-hash result can be used to calculate the jaccard score for measuring the similarity metric. Alternatively, BM25 can be used to determine similarity. On the one hand, a similarity score threshold may not be established during the retrieval of column names—that is, all scores greater than 0 are recorded, and the top "X" highest-scoring entities can be selected. For table values, if the keywords are purely digital, rules that only consider entities that exactly match the keywords can be set (for example, according to a prompt template). For keywords that include both text and numbers, no threshold is applied, and the top five highest-scoring entities are selected. This initial set of entities can provide a short list of entities, which can then be fed back to the reranker for reranking to identify the two most similar entities. The retrieved entities may then be cross-referenced to obtain their corresponding table and column names—that is, the table and column names may be inferred based on the entities obtained during the entity retrieval process at step 212. In one aspect, the final list of entities may also be deduplicated and sorted.

[0064] MinHash(A,B)=Pr(h(A)=h(B)) (1)

[0065]

[0066] Equation 2 represents a measure of similarity calculated using jaccard similarity, where: A, B are two sets, |A∩B| is the size of the intersection between sets A and B, |A∪B| is the size of the union between sets A and B, and the score ranges from 0 (completely different) to 1 (identical). Equation 1 represents the above-mentioned MinHash technique, where: h() is a hash function selected from the hash function family, min(h(A)) is the minimum hash value of all elements in set A, and Pr[] represents the probability that the minimum hash values ​​of two sets are equal is approximately equal to their Jaccard similarity. It should be noted that MinHash can be used to efficiently estimate the Jaccard similarity of large data sets. On the one hand, multiple hash functions can be used to improve accuracy (for example, the more hash functions used, the more accurate the estimate).

[0067] Text description retrieval can cover two types of information: column descriptions and value descriptions. Column descriptions provide additional details about the column name, while value descriptions explain the data within the column, such as how the values ​​are derived. Given the smaller data set in this task, the retrieval method is different from that used in database retrieval. The modeling engine 122 can use an embedding model to encode the data, and then determine the similarity between the two sets of embeddings as described above to calculate the score. These entities can then be re-ranked using a re-ranking model to determine the final order of relevance.

[0068] The query generation process at step 216 (also referred to herein as LAIA-SQL) involves two phases: 1) SQL generation and 2) correction. During SQL generation, in-context learning can be used to provide instructions to the LLM for generating SQL statements. The prompts for this task can be structured into four segments: data schema, user question reasoning, constraints, and incentives. The data schema component can include details such as data format, columns, table names, and examples that integrate entity information obtained during entity retrieval. The user question reasoning segment incorporates the user question, the main task and subtasks identified (e.g., in step 208), and prompts derived from a dataset (e.g., a LAIA-NLU dataset). The query engine 120 can be configured to compile these details into prompts, which can be presented to the LLM to generate an initial SQL statement or query.

[0069] At step 218, an initial query may be executed to query one or more data sources (e.g., Figure 1 one or more databases 118, Figure 1 Data source 140, Figure 1154, or another data source). In one aspect, execution of a query may result in one or more errors. In such cases, the query engine 120 may be configured to implement a correction process to generate a new query. For example, the initial SQL statement may include errors, such as incorrect table names, misaligned columns, or extraneous symbols. To correct these problems, the correction process may pass the erroneous SQL statements back to the LLM along with their corresponding error messages, and the LLM may generate a corrected query that resolves at least some of the errors. The correction process may be an iterative process in which multiple queries are generated and each successive query resolves one or more errors until a syntactically correct and operable SQL query is generated that ultimately produces the correct result (i.e., correctly answers the initial question presented).

[0070] To further illustrate the operation of the query engine 120 and the modeling engine 122, and referring to Figure 3 A block diagram illustrating an exemplary architecture for executing NL2SQL according to the concepts described herein is shown as NL2SQL architecture 300. NL2SQL architecture 300 includes a prompt generator 310, an NLU engine 320, an entity extractor 330, and a query engine 340. In the following description, reference will be made to Figure 2 The operation of the components of the NL2SQL architecture 300 is described with reference to the examples shown in FIG. 3. For example, the prompt generator 310 may receive a prompt from a user (e.g., Figure 1 130). Figure 2 For example, the question could be "What is the highest qualifying free rate for K-12 students in Alameda County schools?"

[0071] As described above, when receiving an NL question as input, the modeling engine 122 can be configured to generate prompts that are provided to the NLU engine 320. The NLU engine 320 can use the prompts to perform NLU-based task decomposition and keyword extraction according to the concepts described herein. In one aspect, the prompts can be generated based on one or more prompt templates. For example, the prompt templates can be as follows:

[0072] You are a professional English teacher.

[0073] Question: {task question}

[0074] 1. The above sentence is completely correct. Please divide the above sentence into main tasks and subtasks.

[0075] 2. Tell me how to implement each subtask and divide it into objects and implementation. You can only detect the key words in the sentence, do not use words that are not included in the sentence.

[0076] 3. The object is related to the keywords in the question.

[0077] 4. The values ​​in the implemented dictionary are mostly one or two words. If the value you selected contains many words, please double-check whether it belongs to the filter criteria and then correct it. It is a number or an adjective.

[0078] ###Example 1:

[0079] {

[0080] 'Question': "Name the schools in Riverside where the average SAT math score is greater than 400. What types of financial aid do these schools offer?", 'Main Task': ["1. Name the schools in Riverside where the average SAT math score is greater than 400", "2. What types of financial aid do these schools offer?"],

[0081] 'Subtask':

[0082] [“1.1 Name the school in Riverside”,

[0083] "1.2 is the average math score of these schools",

[0084] "1.3 Calculate the average score of each school's average math score",

[0085] "1.4 Find schools where the average SAT math score is greater than 400",

[0086] “2.1 Types of funding for these schools”],

[0087] 'object': ['Name the School', 'Aid Type', 'Average SAT Math Score', 'School'],

[0088] 'implemented': [{'in':'Riverside'}, {'greater than':'400'}]

[0089] }

[0090] ###Example 2:

[0091] {

[0092] 'Question': "How many units of item 9 did store 1 sell in January 2012?", 'Main Task': ["Determine the total number of units of item 9 sold in store 1 in January 2012"], 'Subtasks':

[0093] [“1.1 Logo No.1 Store”,

[0094] "1.2 Identification of item No. 9",

[0095] "1.3 Tracking sales in January 2012",

[0096] "1.4 Calculate the total units sold for item 9"],

[0097] 'object': ['unit', 'item number', 'store number'],

[0098] 'implemented': [{'store number': '1'}, {'item number': '9'}, {'in': 'January 2012'}]

[0099] }

[0100] The example prompt template shown above can instruct the LLM of the NLU engine 320 to use the task question (e.g., "What is the highest qualified free rate for K-12 students in Alameda County schools?") into tasks and subtasks as part of the task decomposition process via instructions 1 and 2 of the populated prompt template. Similarly, instruction 3 (i.e., the object is related to the keywords in the question) provides the LLM with instructions to extract the object keywords from the task question, and instruction 4 (i.e., the values ​​in the implemented dictionary are mostly one to two words. If the value you choose contains many words, please confirm again whether it belongs to the filter criteria and then make corrections. It is a number or an adjective.) can be used to improve the accuracy of the keyword extraction stage, such as by providing the LLM with filtering criteria for verifying the extracted keywords. In one aspect, the filtering criteria can be the above-mentioned LAIA-NLU dataset.

[0101] Additionally, the prompt may include one or more examples that can be used to support ICL for task decomposition and keyword extraction. Figure 2 ), the examples included in the prompt generated by the prompt generator 310 have been simplified to indicate only "###Example 1:" and "###Example 2:", but in a real-world implementation may include more information or more or less than two examples.

[0102] Once generated, the modified prompt output by the prompt generator 310 can be provided to the NLU engine 320. The NLU engine 320 can be configured to receive the prompt output by the prompt generator 310 and use the LLM to perform keyword extraction and task decomposition. For example, the LLM can be configured to extract a set of keywords 322 from the question included in the prompt based at least in part on the instructions of the prompt (e.g., only detect keywords in the sentence, do not use words that are not included in the sentence, the object is related to the keywords in the question, and the values ​​in the implemented dictionary are mostly one to two words. If the value you choose contains many words, please confirm again whether it belongs to the filter conditions and then make corrections. It is a number or an adjective), examples, dictionaries, or a combination thereof. In the above example prompt and question, the keywords may include a set of object keywords and a set of implementation keywords as follows: Object: ["Highest Qualified Free Rate", "K-12 Students", "School"]; Implementation: [{"In": "Alameda County"}]. Note that the specific instructions in the example prompts above are provided by way of example and not limitation, and the instructions provided to the LLM may include more or fewer instructions, different instructions, or other instructions that deviate from the above non-limiting examples.

[0103] Additionally, the LLM of the NLU engine 320 may be further configured to decompose the problem into a main task and one or more subtasks (in the example above) based at least in part on the instructions of the prompt (e.g., please divide the sentence above into main tasks and subtasks and tell me how to implement each subtask and divide it into objects and implementations), examples, dictionaries, or a combination thereof, based on the problem included in the prompt. Figure 3 324). In the example prompt and question above, the problem can be broken down into a main task and a set of subtasks as follows:

[0104] Primary Task: [“Identify the schools in Alameda County with the highest qualifying free rates for K-12 students”];

[0105] Subtasks: [“1.1 Identify the schools in Alameda County,” “1.2 Determine the qualifying free rates for K-12 students at these schools,” “1.3 Find the highest qualifying free rate among these schools”].

[0106] As can be understood from the above example, the task decomposition output by the LLM based on the instructions in the prompt accurately identifies the main task of the problem and correctly decomposes the main task into subtasks for implementing the main task. In one aspect, the LLM of the NLU engine 320 can be trained using fine-tuning and other techniques, which can enhance the accuracy of the extracted set of keywords 322 and the accuracy of decomposing the problem into task data 324 output by the LLM. An exemplary process and setting for performing fine-tuning according to the concepts disclosed herein are described in more detail below in the Experimental Setting section.

[0107] As mentioned above, once keyword extraction and task decomposition are completed, the entity retrieval process can be performed. Figure 3 As shown, the entity retrieval process may be performed by an entity extractor 330, which includes an embedding engine 332, an entity retriever 334, and a reranker 336. The embedding engine 332 may be configured to generate a set of keyword embeddings based on the set of keywords 322 and a set of table embeddings based on the table data 352. The table data 352 may include tables maintained in one or more data sources 350 (e.g., included in Figure 1 As mentioned above, Figure 2 As explained in step 210 of , the set of table embeddings generated based on the table data 352 can encode the pattern of one or more tables into a form that can be used to detect the portion of the one or more tables related to the keywords included in the set of keywords 322. Similarly, the set of keyword embeddings can encode the set of keywords 322 into the same form as the set of table embeddings. In one aspect, the embeddings can be generated using MinHash, Stella-400M, Stella-1.5B, text-embedding-3-large, or other embedding techniques and algorithms.

[0108] The entity retriever 334 may be configured to receive the set of table embeddings and the set of keyword embeddings and identify entities based on comparing the two sets of embeddings. For example, the entity retriever may be configured to identify table names, column names, rows, or other information in the table embeddings (and corresponding elements of the table data 352) that are similar or related to one or more keywords included in the set of keyword embeddings. In one aspect, the entity retriever 334 may limit the identified entities to a threshold number of entities. For example, the entity retriever 334 may limit the number of identified entities to the first 10 entities (e.g., for each keyword).

[0109] The set of entities identified by the entity retriever 334 may be provided to a reranker 336. The reranker 336 may be configured to rank the entities and output a final set of entities based on the rankings. For example, the reranker 336 may rank the entities and then output a subset of the entities for query generation. In one aspect, the reranker 336 ranks the entities based on one or more criteria. The ranking criteria may be learned by the reranker 336 through training, fine-tuning, or other techniques. In one aspect, the reranking model may be pre-trained with a large data set. Note that this may take more time to process the big data embedding job than the retriever, but pre-training may achieve better performance. In one aspect, the ranking may be obtained based on a similarity metric calculated by the reranker 336. In one aspect, the similarity metric may be cosine similarity. In additional or alternative aspects, the reranker 336 may use other similarity metrics to rank the entities output by the entity retriever 334. Note that entity retriever 334 may be configured to identify a subset of data (e.g., elements of tabular data 352 that are relevant to the set of keywords) from a larger data set (e.g., tabular data 352 that may include many tables, column names, rows and values, metadata, etc.). Thus, by using a separate ranking process provided by re-ranker 336, a more accurate ranking may be obtained, which may contribute to the accuracy of one or more queries generated based on the extracted entities.

[0110] After ranking the extracted entities, the ranked entity list or a subset of the ranked entity list (e.g., the top "X" entities) may be provided to the query engine 340. In one aspect, the query engine 340 may be Figure 1 324. As described above, the query engine 340 may be configured to generate one or more query commands (e.g., SQL statements) based on the ranked entities and the task data 324. The query engine 340 may utilize the LLM 342 to generate query commands based on the task data 324 and the ranked entities output by the entity extractor 330. In one aspect, the query engine 340 may be configured to incorporate the task data 324 and the ranked entities into a prompt that may be passed to the LLM 342. The prompt may include instructions for generating query commands, which may be included in a prompt template to which the task data 324 and the ranked entities may be added. In one aspect, the prompt for query command generation may be as follows:

[0111] You are a data science expert.

[0112] Below, you are presented with the database schema and the problem.

[0113] Your task is to read the schema, understand the question, and generate an efficient SQLite query to answer the question. Think through how to write the query step by step before generating the final SQL query.

[0114] Database Schema

[0115] {Database Schema}

[0116] The schema provides an in-depth description of the database's schema, detailing the tables, columns, primary keys, foreign keys, and any relevant information about relationships or constraints.

[0117] NOTE!!! Special attention should be paid to the examples listed next to each column of the data schema, as they directly hint at which columns are relevant to our query.

[0118] ###constraint

[0119] 1. For the key phrases mentioned in the question, we provide the most similar values ​​within the column indicated by "--example" before the corresponding column name. This is the key hint to indicate the correct column to use for your SQL query.

[0120] 2. Note!!! Avoid using different columns for the same object with different filter values.

[0121] 3. Attention!!! Do not write the wrong column in the SQL code. Please check again in SQL whether the column belongs to the table.

[0122] ###question:

[0123] {question}

[0124] ###Steps you should follow:

[0125] {Main Task}

[0126] {Subtask}

[0127] {hint}

[0128] The main task, subtasks and evidence are correct, please generate the final SQL query based on them, please strictly follow the main task, subtasks and evidence.

[0129] If there is an equation in the proof, follow it exactly!!!

[0130] The amount of items SELECT in the sql query depends on the number of master tasks. If there is only one master task, you should select only one item related to the master task in the sql query.

[0131] Please respond with a JSON object constructed as follows:

[0132] {{"SQL":"Your SQL query is here."}}

[0133] As shown above, the hint incorporates information about the schema of the database to be queried, the question presented, the task decomposition data 354, and a ranked list of retrieved entities can be provided as an example. Additionally, the hint incorporates constraints that help the LLM map keywords to retrieved entities during the generation of the query command. Figure 2 In the example, the query command generated by LLM342 based on the above prompts may be:

[0134] Second SQL: SELECT`Percent(%)Eligible Free(K-12)`FROM FRPM WHERE`CountyName`='Alameda'AND`Percent(%)Eligible Free(K-12)`IS NOT NULL ORDER BY`Percent(%)Eligible Free(K-12)`DESC LIMIT 1

[0135] As can be understood from the example query above, the subtask "Identify schools in Alameda County" can be incorporated into the query command via "WHERE 'County Name' = 'Alameda'". Similarly, the subtask "Determine the eligible free rate for K-12 students at these schools" can be reflected in AND 'Percent (%) Eligible Free (K-12)' IS NOT NULL". It should be noted that the improved task decomposition achieved using the techniques disclosed herein can improve the accuracy of the generated query commands. For example, by identifying a main task and subtasks used to implement the main task, the query generator can more accurately create a query command designed to implement the main task based on the subtasks.

[0136] Once the query command is generated, the query command may be executed by the query logic 346 against the data source 350 to retrieve data based on the query output by the LLM 342. In one aspect, the initial query command output by the LLM 342 may contain an error, and the query execution may fail due to the error. For illustration, executing the example query above may result in the display of an error message of "ERRORINFORMATION: FRPM table is not found". If the query logic 346 detects that an error is received in response to executing the query command against the data source 350, the query logic 346 may pass the query command and the error response to the correction engine 344. The correction engine 344 may be configured to generate a new prompt to the LLM 342, the new prompt containing the initial query command and the error response and instructions to rewrite the query command to fix the error. For example, the prompt for correcting the query command may be as follows:

[0137] Purpose: Your purpose is to ensure that the query follows the database administration instructions and uses the correct conditions.

[0138] Database Schema:

[0139] {Database Schema}

[0140] ###constraint

[0141] 1. When you need to find the highest or lowest value based on a condition, using ORDER BY + LIMIT 1 in a subquery is better than using MAX / MIN.

[0142] 2. If the predicted query includes an ORDER BY clause to sort the results, you should include these columns in the SELECT clause only if the problem explicitly requires the columns to be used for sorting. Otherwise, omit these columns from the SELECT.

[0143] 3. The predicted query should return all the information asked in the question without any missing or additional information.

[0144] 4. For the key phrases mentioned in the question, we provide the most similar values ​​within the column indicated by "--Example" before the corresponding column name. This is the key hint to indicate the correct column to use for your SQL query.

[0145] 5. If you are joining multiple tables, make sure to use aliases for the tables and use these aliases to refer to the columns in the query. Use T1, T2, T3, ... as aliases.

[0146] ###question:

[0147] {question}

[0148] ###error message

[0149] {error message}

[0150] ###Steps you should follow:

[0151] {Main Task}

[0152] {Subtask}

[0153] {hint}

[0154] ###Predicted query:

[0155] {SQL}

[0156] Pay attention to the ERROR INFORMATION and modify the SQL query based on the error.

[0157] Consider whether the predicted query already uses hints and evidences, and if not, use the hints and evidences in SQL query generation.

[0158] Please respond with a JSON object constructed as follows (if the SQL query is correct, return the query as is):

[0159] {{"revised_SQL":"Your revised SQL query is here."}}

[0160] As shown above, the correction hint combines instructions on how to correct the SQL query based on error information, previous queries, task decomposition data 334 and other information (e.g., constraints) to the LLM to help the LLM correct the query. The LLM 342 can receive the correction hint and generate a new query command. For example, after the first correction iteration, the above query command can be rewritten to correct the table name "FRPM" to the correct table name "frpm", which can produce the following query command:

[0161] Second SQL: SELECT`Percent(%)Eligible Free(K-12)`FROM frpm WHERE`CountyName`='Alameda'AND`Percent(%)Eligible Free(K-12)`IS NOT NULL ORDER BY`Percent(%)Eligible Free(K-12)`DESC LIMIT 1

[0162] Executing the corrected query command may return the correct data to answer the question, and the data may be transmitted or presented to the user as output data 348. Note that the correction process may be performed iteratively, with each successive iteration correcting zero or more errors. In one aspect, the query logic 346 or correction engine 344 may be configured to limit the number of iterations that may be performed to a threshold number of iterations (e.g., 5 iterations). This may prevent an infinite loop from occurring where errors cannot be automatically corrected by the correction engine 344.

[0163] Output data 348 may be presented to a user via a graphical user interface, such as a browser-based interface displayed in a web browser (e.g., of computing device 130), a web browser (e.g., of computing device 130), or a graphical user interface (e.g., of computing device 130). Figure 1The output data 348 may be processed using an application interface (such as an application running on computing device 130), a display of a mobile device, or other type of graphical user interface. In one aspect, the output data 348 may be processed using the LLM before being transmitted to the user. For example, assume that the highest qualifying free rate obtained using the query above is [1.0]. Instead of simply outputting 1.0, the output data 348 may be passed to the LLM 342 along with the original question presented, and a natural language sentence answering the question may be created and passed to the user. For example, the output data 348 may be returned to the user as "The highest qualifying free rate for K-12 students in schools in Alameda County is 1.0."

[0164] Note that LLM 342 may be the same LLM used for keyword extraction and task decomposition. In additional or alternative implementations, architecture 300 may utilize separate LLMs for NLU tasks (e.g., keyword extraction and task decomposition) and query generation. As can be appreciated from the above description, architecture 300 provides an improved method for performing NL2SQL query command generation via an enhanced keyword extraction and task decomposition process utilizing intelligent hint engineering and a tabular QA dataset specifically designed to enhance the ability of the LLM to perform NLU tasks during keyword extraction and task decomposition. Similarly, the task decomposition and entity retrieval process may be used to improve the manner in which the LLM may be used to generate query commands (e.g., SQL statements) that are more likely to retrieve relevant data and are more easily able to handle multi-table queries. Thus, one of ordinary skill in the art will recognize that, as described with reference to Figure 3 The described architecture 300 and its supported operations provide improvements to the NL2SQL domain and systems supporting table QA functionality.

[0165] To further illustrate the improvements provided by the concepts disclosed herein, experimental results will now be described.

[0166] Experimental results show that LAIA-SQL (for example, Figure 1 System 100, Figure 2 The process of 200 and Figure 3The method and functionality provided by the architecture of LAIA-SQL outperform all prior art open source NL2SQL methods, achieving 67.28% accuracy on the BIRD dev dataset and 88.7% accuracy on the Spider dev dataset. The experimental results also show that LAIA-SQL also has fairly fast processing, answering 10 questions in only 56.81 seconds at a cost of $0.32 with 80% accuracy. Compared to the leading NL2SQL method using GPT-4o, LAIA-SQL reduced the runtime by 52.4% and reduced the runtime cost by 97%, while maintaining the highest accuracy among advanced open source NL2SQL methods. Therefore, it should be understood that LAIA-SQL reduces energy consumption due to the reduced runtime, while also reducing the cost of utilizing NL2SQL and improving accuracy, thereby making NL2SQL technology more feasible for real-world applications and uses.

[0167] In order to rigorously evaluate LAIA-SQL, a series of comprehensive experiments were conducted. These comprehensive experiments include a comparison of SQL generation accuracy on Bird and Spider datasets relative to SOTA NLSQL methods. Additionally, the practical utility of LAIA-SQL is evaluated relative to the leading open source NL2SQL method. Ablation studies are also conducted to examine the contributions of different models and modules within the LAIA-SQL architecture. The performance of various models is evaluated from multiple perspectives based on fine-tuning using the LAIA-NLU dataset. Collectively, these experiments provide a multi-faceted evaluation of the effectiveness of LAIA-SQL. The results of these experiments are described and shown in more detail below.

[0168] Table 1 below shows the performance on the Bird and spider datasets.

[0169] Table 1

[0170]

[0171] Table 2 below shows the actual utility metrics of the NL2SQL approach using GPT-4o as the base model.

[0172] Table 2

[0173]

[0174] Experimental setup

[0175] As a baseline, NL2SQL methods that are open source or have been published in papers and include GPT-4 as a baseline model were selected. The selected methods are as follows: Distillery Maamari et al. (2024), which uses the pattern linking amplification technique; CHESS Talaei et al. (2024), which integrates data catalogs and database values ​​for SQL generation; MAC-SQL Wang et al. (2023), which features a multi-agent collaboration framework; Dail-SQL Gao et al. (2023), which combines prompt engineering with question representation, example selection and organization; and CodeS-15B Li et al. (2024b), which uses an incremental pre-training method on a curated SQL-centric corpus.

[0176] In the user question understanding module (for example, Figure 3 In the NLU engine 320 of , various models such as GPT-4o-mini OpenAI (2024a), GPT-4Achiam et al. (2023), Mistral-7B Jiang et al. (2023), LLaMA3-8B Dubey et al. (2024), Baichuan2-7B and 13B Yang et al. (2023) were evaluated. For the entity retrieval module (e.g., Figure 3 For entity retrievers 330), the performance of MinHash combined with Jaccard Score was compared to BM25 Robertson et al. (2009) for the retriever. For embedding models (e.g., embedding engine 332), text-embedding-3-large OpenAI (2024b), Stella-1.5B, and Stella-400M were evaluated. For code generation models (e.g., Figure 3 During the fine-tuning phase of the query generator 340), DeepSeek-Coder-V2-Instruct, DeepSeek-Coder-V2-Base, and Qwen-1.5-Coder were evaluated.

[0177] Fine-tuning was performed using a setup of 4 Nvidia 4090 GPUs and leveraged distributed data parallelism and DeepSpeed. A uniform batch size of 1 was maintained and the epoch count was set to 1. The learning rate was fixed at 2e -4 Additionally, the Low Rank Adaptation (LoRA) Hu et al. (2021) technique is utilized, with specific parameters: LoRA rank is 64, LoRA alpha is 16, and dropout probability is 0.05. The bit accuracy is set to 4. Note that fine-tuning the LAIA-NLUer model takes about 30 minutes, while the code generation model takes 4 to 5 hours.

[0178] The evaluation of the task decomposition in NLU is evaluated based on the quality of the generated reasoning results relative to the human-labeled ground truth results using BLEU, ROUGE, and GPT-4o scores. Specifically, BLEU-1 and BLEU-2 provide insights into language accuracy by measuring the n-gram match between the generated description and the ground truth. ROUGE-1, ROUGE-2, and ROUGE-L are used to evaluate the overlap of n-grams, sequences, and word pairs, providing a measure of the comprehensiveness and relevance of the description. Additionally, a five-point Likert scale table conducted by GPT-4o is used to measure the overall quality and similarity to human annotations.

[0179] For the keyword extraction task in NLU, the performance of the model was evaluated using precision, recall, and finally the F1 score. These metrics provide a balance between the correctness of the extracted keywords and the recall ability of the model, thus providing an overall view of its extraction efficiency.

[0180] Execution accuracy (Ex) is used to measure the correctness of SQL queries by comparing the results of the predicted query executed with the reference query on a specific database instance. This metric not only ensures semantic correctness but also takes into account the variations of the SQL formula that produce the same results.

[0181] Table 3 below shows the results of the module ablation study of LAIA-SQL on the development set of the Bird dataset.

[0182] Table 3

[0183]

[0184]

[0185] Table 4 below shows the results of the model ablation study of LAIA-SQL on the development set of the Bird dataset.

[0186] Table 4

[0187]

[0188] For the BIRD dataset, due to its anonymity policy, only the execution accuracy on the development dataset is reported. As shown in Table 1, LAIA-SQL obtains the best Dev EX compared to other state-of-the-art models and is also the best open source method currently available. In the Spider dataset, LAIA-SQL shows the highest execution accuracy in both the development and test datasets compared to all other state-of-the-art models.

[0189] Additionally, according to the actual value evaluation in Table 2, it can be observed that LAIA-SQL performs best in aspects such as time efficiency, operation cost, and accuracy. Compared with the best open source method CHESS, LAIA-SQL achieves a 52.4% reduction in running time and a 97% reduction in running cost, demonstrating meaningful industrial application potential. Overall, LAIA-SQL remains the highest performing method among the open source NL2SQL methods.

[0190] As shown in Table 3, in the ablation study, significant improvements in accuracy were observed for each additional module. Notably, LAIA-NLUer, designed for keyword extraction and task decomposition, achieved the highest accuracy increase, with an improvement of 7.66% over the previous method. The entity retrieval module also showed significant gains, with an accuracy improvement of 4.9%. Overall, LAIA-NLUer, entity retrieval, and revision modules each contributed to the accuracy improvement.

[0191] The results of the model ablation study shown in Table 4 show that in the entity retrieval module, MinHash outperforms BM25, with an accuracy improvement of two percentage points and consumes only one-third of the time taken by BM25. Additionally, different performances are observed among different embedding models. Unexpectedly, the stell-400M model outperforms the stella-1.5B model, indicating that larger parameter models do not necessarily produce better embedding results.

[0192] In the code generation module, the base model was compared to a fine-tuned version, and it was found that the fine-tuned model did not perform as well as GPT-4. However, it is important to note that the chosen model has only 22 billion parameters, which shows that the number of parameters especially affects the accuracy of the model for complex tasks such as code generation.

[0193] Supervised Fine-tuning As shown in Table 5, we find that large and small models are suitable for different fine-tuning tasks. For example, large models such as GPT-4 and GPT-4o-mini show significantly better performance on complex tasks such as task decomposition after fine-tuning compared to smaller models. However, for tasks that do not require deep understanding, such as keyword extraction, smaller models such as Mistral-7B outperform larger models. Overall, our findings suggest that the decision to use large or small models for fine-tuning should be guided by the specific requirements of the task, as the performance of fine-tuned large models is not universally outstanding.

[0194] Table 5 below shows the comparison of the fine-tuned models in task decomposition and keyword extraction.

[0195] Table 5

[0196]

[0197] Table 6 below shows the impact of dataset size and epochs on the performance of LAIA-NLU in terms of F1 score.

[0198] Table 6

[0199]

[0200] In addition, the effects of different dataset sizes and different epochs on the fine-tuning performance of keyword extraction are compared. In Table 6, it is shown that the Mistral-7B model has the best overall performance, followed by the LLaMA-8B model. It is worth noting that it can be observed that for all models except Mistral-7B, the F1 score initially increases with the increase in training data size and then decreases. This shows that more data is not always better. In addition, it is found that increasing the number of epochs greatly improves the F1 score, indicating that adding more epochs is the most effective way to improve the accuracy of keyword extraction.

[0201] Although the disclosed techniques and models outperform many prior art NL2SQL approaches, their accuracy can be further enhanced. For example, fine-tuning the specific dataset on which the query commands are to be executed can improve performance and make the disclosed techniques more general across different domains. Computational limitations restrict the models used in the experiments to smaller models; however, larger models such as DeepSeek-V2-Coder-236B and Llama3.1-70B can be easily trained and fine-tuned using the concepts described herein to potentially provide performance that is superior to the current 22B model used in the experiments, which can significantly improve accuracy. In addition, the entity retrieval component of the tested LAIA-SQL setting uses Min-Hash with Jaccard Score and BM25, but utilizing an advanced RAG module can enhance this aspect and improve the entity retrieval process. In addition, due to resource constraints, the proof-of-concept LAIA-NLU dataset is limited to 1500 samples, affecting the robustness of the LAIA-NLUer model.

[0202] As shown above, Figure 1 System 100, Figure 2 The process of 200 and Figure 3The architecture is based on the development of the LAIA-NLU dataset and the retrieval-enhanced NL2SQL framework, LAIA-SQL introduces significant advances in table QA methods. LAIA-NLU contains 1,500 high-quality instructions, enabling the training of LAIA-NLUer, a pioneering NLU model customized for table QA. By integrating LAIA-NLUer, the disclosed NL2SQL method (referred to as LAIA-SQL in this article) shows significant improvements over all existing models and pipelines, achieving 67.28% higher accuracy for 10 questions and reducing SQL query execution time by 52.4% to 56.81 seconds. At the same time, the cost of a question is reduced to 0.032USD. These findings highlight the potential of the disclosed method to enhance the efficiency and accuracy of multi-table data retrieval, making it more accessible to non-expert users.

[0203] refer to Figure 4 , a flowchart of a method for executing a query using NL2SQL technology according to aspects of the present disclosure is shown as method 400. In one aspect, the steps of method 400 may be stored as instructions (e.g., Figure 1 Instructions 116), which are executed by one or more processors (e.g., Figure 1 When executed by one or more processors 112 of the system, the one or more processors perform the steps of method 400. In one aspect, method 400 may be performed in a system such as Figure 1 The system can be designed to perform the steps of method 400 to perform Figure 2 The operations of process 200 may include: Figure 3 The query engine of the architecture 300 (e.g., Figure 1 and 2 The query engine 120 and Figure 3 query engine 340) and a modeling engine (e.g., Figure 1 The modeling engine 122 and Figure 3 corresponding elements of the modeling engine shown).

[0204] At step 410, method 400 includes receiving, by one or more processors, a natural language question associated with information stored in at least one table in a database. At step 420, method 400 includes generating, by one or more processors, a prompt based on the natural language question. As described above, the prompt may include: a first instruction corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question; and a second instruction corresponding to a task decomposition process configured to determine one or more tasks for answering the question. At step 430, method 400 includes executing, by one or more processors, a keyword extraction process and a task decomposition process via an LLM based on the first instruction and the second instruction of the prompt to generate a set of keywords and task decomposition data. In one aspect, the set of keywords may be Figure 3 A set of keywords 322, and the task decomposition data can be Figure 3 Task decomposition data 324.

[0205] At step 440, method 400 includes retrieving, by one or more processors, a set of entities from the at least one table based on the set of keywords. In one aspect, the set of entities may be retrieved and ranked using an entity retrieval process, as described above with reference to Figure 2 as well as Figure 3 At step 450, method 400 includes generating, by the one or more processors, a second prompt based on the set of entities, the task decomposition data, and the question. In one aspect, the second prompt includes a third instruction for generating a query statement to extract an answer to the question from the at least one table, as described above with reference to Figure 3 As described by the query engine 340 of . At step 460, the method 400 includes generating a query statement via the LLM based on the second prompt by one or more processors. At step 470, the method 400 includes executing the query statement against the at least one table by one or more processors to obtain the answer to the question from the at least one table. In one aspect, the execution of the query may result in one or more errors. As described above, a correction process may be performed to generate new prompts for query generation to fix one or more of these errors. The correction process may be iteratively performed until the query is successfully executed or a stopping criterion is reached. After the query is successfully executed, the result of the query may be output to a user device (e.g., Figure 1 It should be noted that the steps of method 400 may include the following: Figures 1 to 3 Additional steps and functions described.

[0206] Although the embodiments of the present disclosure and advantages thereof have been described in detail, it should be understood that various changes, substitutions and variations may be made herein without departing from the spirit and scope of the present disclosure as defined by the appended claims. In addition, the scope of the present application is not intended to be limited to the specific embodiments of the processes, machines, manufactures, material compositions, devices, methods and steps described in this specification. As will be readily understood by those of ordinary skill in the art from the present disclosure, according to the present disclosure, it is possible to utilize the processes, machines, manufactures, material compositions, devices, methods or steps currently existing or to be developed later that perform functions substantially the same as those in the corresponding embodiments described herein or realize results substantially the same as those in the corresponding embodiments described herein. Therefore, the appended claims are intended to include such processes, machines, manufactures, material compositions, devices, methods or steps within their scope.

Claims

1. A system for executing queries using natural language to structured query language (NL2SQL), the system comprising: Memory; one or more processors communicatively coupled to the memory and configured to: receiving a natural language question associated with information stored in at least one table of a database; Generate a prompt based on the natural language question, wherein the prompt includes: first instructions corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question; and second instructions corresponding to a task decomposition process configured to determine one or more tasks for answering the question; The first instruction and the second instruction based on the prompt perform the keyword extraction process and the task decomposition process via a large language model (LLM) to generate a set of keyword and task decomposition data; retrieving a set of entities from the at least one table based on the set of keywords; and generating a second prompt based on the set of entities, the task decomposition data, and the question, wherein the second prompt includes third instructions for generating a query statement to extract an answer to the question from the at least one table; generating the query statement via the LLM based on the second prompt; and The query statement is executed against the at least one table to obtain the answer to the question from the at least one table.

2. The system of claim 1, wherein the LLM is fine-tuned based on a dataset comprising a plurality of question and query pairs, each question and query pair comprising a sample natural language question and a ground truth query corresponding to the sample natural language question.

3. The system of claim 1 , wherein the one or more processors are configured to: receiving a query result in response to executing the query statement; and The query result is output, wherein the query result includes the answer to the question from the at least one table.

4. The system of claim 3, wherein the one or more processors are configured to: determining whether the query result includes an error; and A remediation process configured to generate a remediation query statement is executed, wherein the remediation query statement addresses at least a portion of the error. The system of claim 4 , wherein the correction process is performed iteratively until a stopping criterion is reached. 6 . The system of claim 1 , wherein the task decomposition process is configured to determine a main task and one or more subtasks for implementing the main task.

7. A system according to claim 6, wherein the second instruction includes one or more examples, and each of the one or more examples includes a natural language question, a main task derived from the natural language question, and one or more subtasks for implementing the main task derived from the natural language question. The system of claim 1 , wherein the set of keywords includes object keywords and implementation keywords.

9. The system of claim 1, wherein the third instructions of the second prompt include information associated with the set of entities, a data schema of the at least one table of the database, the problem, and the task decomposition data.

10. The system of claim 1, wherein the one or more processors are configured to: generating a set of keyword embeddings based on the set of keywords and generating a set of table embeddings based on the data schema; and An initial set of entities is determined based at least in part on the set of keyword embeddings and the set of table embeddings.

11. The system of claim 10, wherein the set of entities is determined based at least in part on a ranking of an initial set of entities.

12. A method for executing a query using Natural Language to Structured Query Language (NL2SQL), the method comprising: receiving, by one or more processors, a natural language question associated with information stored in at least one table of a database; Generating, by the one or more processors, a prompt based on the natural language question, wherein the prompt comprises: first instructions corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question; and second instructions corresponding to a task decomposition process configured to determine one or more tasks for answering the question; executing, by the one or more processors, the keyword extraction process and the task decomposition process via a large language model (LLM) based on the first instruction and the second instruction of the prompt to generate a set of keyword and task decomposition data; retrieving, by the one or more processors, a set of entities from the at least one table based on the set of keywords; generating, by the one or more processors, a second prompt based on the set of entities, the task decomposition data, and the question, wherein the second prompt includes third instructions for generating a query statement to extract an answer to the question from the at least one table; generating, by the one or more processors via the LLM, the query statement based on the second hint; and The one or more processors execute the query statement against the at least one table to obtain the answer to the question from the at least one table.

13. The method of claim 12, further comprising fine-tuning the LLM based on a dataset comprising a plurality of question and query pairs, each question and query pair comprising a sample natural language question and a ground truth query corresponding to the sample natural language question.

14. The method of claim 12, wherein the one or more processors are configured to: receiving a query result in response to executing the query statement; and The query result is output, wherein the query result includes the answer to the question from the at least one table.

15. The method according to claim 14, further comprising: determining whether the query result includes an error; as well as A revising process configured to generate a revised query statement is performed, wherein the revised query statement addresses at least a portion of the error, wherein the revising process is iteratively performed until a stopping criterion is reached.

16. A method according to claim 12, wherein the task decomposition process is configured to determine a main task and one or more subtasks for implementing the main task, wherein the second instruction includes one or more examples, and wherein each of the one or more examples includes a natural language problem, a main task derived from the natural language problem, and one or more subtasks for implementing the main task derived from the natural language problem.

17. The method of claim 12, wherein the set of keywords includes object keywords and implementation keywords, and wherein the first instructions include instructions for identifying, by the LLM, the object keywords and the implementation keywords from the problem.

18. The method of claim 12, wherein the third instructions of the second prompt include information associated with the set of entities, a data schema of the at least one table of the database, the problem, and the task decomposition data.

19. The method according to claim 12, further comprising: generating a set of keyword embeddings based on the set of keywords and generating a set of table embeddings based on the data schema; determining an initial set of entities based at least in part on the set of keyword embeddings and the set of table embeddings; as well as The set of entities is ranked to produce a ranked set of entities, and wherein the query statement is generated based at least in part on the ranked set of entities.

20. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations to execute a query using Natural Language to Structured Query Language (NL2SQL), the operations comprising: receiving, by one or more processors, a natural language question associated with information stored in at least one table of a database; Generating, by the one or more processors, a prompt based on the natural language question, wherein the prompt comprises: first instructions corresponding to a keyword extraction process configured to extract one or more keywords from the natural language question; and second instructions corresponding to a task decomposition process configured to determine one or more tasks for answering the question; executing, by the one or more processors, the keyword extraction process and the task decomposition process via a large language model (LLM) based on the first instruction and the second instruction of the prompt to generate a set of keyword and task decomposition data; retrieving, by the one or more processors, a set of entities from the at least one table based on the set of keywords; generating, by the one or more processors, a second prompt based on the set of entities, the task decomposition data, and the question, wherein the second prompt includes third instructions for generating a query statement to extract an answer to the question from the at least one table; generating, by the one or more processors via the LLM, the query statement based on the second hint; and The one or more processors execute the query statement against the at least one table to obtain the answer to the question from the at least one table.

Images