NL2SQL large model training data synthesis method and system based on error feedback and storage medium
By identifying entities and optimizing NL-SQL question-answer pair generation using RAG technology and error feedback mechanisms, the problem of inaccurate generation in existing technologies is solved, and high-quality NL-SQL question-answer pairs are automatically generated, improving the efficiency and accuracy of the system.
Patent Information
- Application Number
- CN202510977089.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-16
- Publication Date
- 2025-11-25
AI Technical Summary
Existing technologies suffer from problems such as incoherent natural language questions, SQL syntax errors, or mismatches when generating NL-SQL question-answer pairs. Furthermore, manual generation methods are time-consuming and costly, making it difficult to meet the needs of large-scale Text2SQL systems.
By identifying entities in seed question-answer pairs, RAG technology is used to match relevant knowledge in the knowledge base, generate SQL queries and convert them into natural language questions, and combine them with an error feedback mechanism for quality assessment and optimization. Errors are detected and fed back in stages to improve the quality of generation.
The generated NL-SQL question-answer pairs have high logical reasoning complexity, improved fluency and comprehensibility of natural language questions, and enhanced accuracy and question relevance of SQL statements, reducing labor costs and human error.
Smart Images
Figure CN121009104A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a NL2SQL large model training data synthesis method and system based on error feedback and a storage medium. BACKGROUND
[0002] Text2SQL refers to converting natural language (NL) questions in the database field into structured query language (SQL) that can be executed in a relational database. Text2SQL allows users to query the database using natural language, thereby reducing the learning cost of users for professional structured language and significantly improving query efficiency.
[0003] In recent years, the outstanding performance of large language models (LLMs) in the field of natural language processing, especially in strong language understanding ability for NL questions and strong code generation ability for SQL statements, has made them gradually become the key tool for solving Text2SQL problems. Many Text2SQL frameworks based on large language models have been proposed and practiced.
[0004] Text2SQL frameworks based on large language models require high-quality NL-SQL question and answer pair data. High-quality question and answer pair data should have accuracy, diversity, and representativeness. These data can be used not only in the training stage of large language models to improve their performance, but also in the performance evaluation of Text2SQL frameworks. However, such data is difficult to collect through automated means, and currently high-quality NL-SQL question and answer pairs mostly need to be written manually.
[0005] The data quality of NL-SQL question and answer pairs is crucial for the performance improvement and performance evaluation of Text2SQL systems based on large language models. Current research on the generation method of high-quality NL-SQL question and answer pairs mainly focuses on manual generation, which has the disadvantages of high cost, long time consumption, and resource scarcity. In contrast, using large language models to automatically generate NL-SQL question and answer pairs can not only reduce labor costs and reduce human errors, but also rapidly expand the data scale to meet the demand of some large-scale Text2SQL systems for the number of training question and answer pairs.
[0006] However, although large language models have shown great potential in the automatic generation of NL-SQL question and answer pairs, current methods still face multiple challenges. These methods often produce unnatural and difficult-to-understand natural language questions, and there are syntax errors or mismatches with the original question in the SQL statements. SUMMARY
[0007] To solve the problems in the prior art, the present application provides a NL2SQL large model training data synthesis method based on error feedback, comprising the following steps:
[0008] Step 1: identify the entities in the seed question and answer pair, which include the Schema area in the database and the entities in natural language;
[0009] Step 2: use RAG technology to match the knowledge related to the question and entity in the knowledge base;
[0010] Step 3: generate the corresponding SQL query according to the obtained knowledge and entity information, and convert it into a natural language question;
[0011] Step 4: quality evaluation is performed on the generated SQL question and answer pair to ensure that high-quality NL-SQL question and answer pairs are added to the training set, and NL-SQL question and answer pairs with errors are fed back to the NL-SQL question and answer generation link to avoid the repeated occurrence of the same errors.
[0012] As a further improvement of the present application, in the step 1, the entity recognition is divided into two parts: one part is to identify the database tables, columns and values involved in the input SQL; the other part is to identify the entities in the question.
[0013] As a further improvement of the present application, in the step 1, it further comprises:
[0014] Step S1: use the spaCy library in Python to perform word segmentation and entity recognition on the input natural language question;
[0015] Step S2: extract the tables and fields involved from the given SQL query through regular expressions.
[0016] As a further improvement of the present application, in the step 2, it further comprises:
[0017] Step Y1: based on the input natural language question and database Schema information, retrieve the background information related to the task from the external knowledge base;
[0018] Step Y2: use natural language processing technology to convert the question into an embedding vector, and find similar entries in the knowledge base through vector retrieval technology.
[0019] As a further improvement of the present application, in the step 3, the SQL synthesizer and the NL generator are responsible for the generation of SQL statements and the translation of natural language queries, while the feedback mechanism of the error memory library is used to optimize the generation quality.
[0020] As a further improvement of the present application, the SQL synthesizer is responsible for combining database Schema information and external knowledge to generate candidate SQL queries, which not only contain tables and fields from the Schema, but also incorporate relevant information from the external knowledge base.
[0021] The NL generator is responsible for receiving the SQL statements output by the SQL synthesizer and converting them into natural language questions, forming NL pairs with the SQL statements.
[0022] As a further improvement of the present application, in step 3, the temperature parameter method of adjusting the reasoning model is adopted to ensure the diversity of SQL query generation.
[0023] As a further improvement of the present application, in step 4, it also includes:
[0024] Error detection step: In the error detection link, the synthesized data needs to go through three detection processes: syntax detection, execution detection and NL semantic detection; In syntax detection, SQL statements are parsed with the help of sqlparse and sqlglot libraries, and try statement blocks are used to capture syntax abnormalities; Execution detection is performed by connecting the database through the sqlite library, and try statement blocks are also used to capture runtime errors; NL semantic detection uses large language models to score question and answer pairs;
[0025] Error feedback step: In the error feedback stage, different types of error data are fed back to the corresponding links of the synthesis process; SQL syntax errors and SQL execution errors are fed back to the SQL synthesis stage to optimize the generation quality of SQL statements; NL semantic errors are fed back to the NL generation stage to improve the semantic accuracy of natural language queries.
[0026] The present application also discloses a NL2SQL large model training data synthesis system based on error feedback, comprising a memory, a processor and a computer program stored in the memory, the computer program being configured to realize the steps of the method of the present application when called by the processor.
[0027] The present application also discloses a computer readable storage medium, which stores a computer program configured to realize the steps of the method of the present application when called by a processor.
[0028] The present application has the beneficial effect that the method of the present application not only improves the fluency and understandability of natural language questions while ensuring that the generated NL-SQL question and answer pairs have a certain logical reasoning complexity, but also ensures the accuracy of the generated SQL statements and the compatibility with the questions. BRIEF DESCRIPTION OF DRAWINGS
[0029] Figure 1 is the Text-to-SQL data synthesis framework of the present application. DETAILED DESCRIPTION
[0030] Glossary:
[0031] Schema area: Generally refers to the logical area in a database used to define and manage data structures (tables, fields, relationships, etc.).
[0032] spaCy library: A popular open-source library for natural language processing (NLP).
[0033] sqlite library: A lightweight, embedded Relational Database Management System (RDBMS) whose core library (sqlite3) is directly integrated into the application without the need for a separate server process.
[0034] For a single question and answer pair in the seed data set, the Text-to-SQL data synthesis framework of the present application is mainly as shown in Figure 1 .
[0035] The present application discloses an NL2SQL large model training data synthesis method based on error feedback, comprising:
[0036] Step 1: Identify entities in Seed Data Pairs, including Schema areas in the database and entities in natural language;
[0037] Step 2: Use RAG technology to match knowledge related to the question and entity in the knowledge base;
[0038] Step 3: Generate the corresponding SQL query according to the obtained knowledge and entity information, and convert it into a natural language question;
[0039] Step 4: Quality assessment of the generated SQL question and answer pair to ensure that high-quality NL-SQL question and answer pairs are added to the training set, while NL-SQL question and answer pairs with errors are fed back to the NL-SQL question and answer generation link to avoid the repeated occurrence of the same errors.
[0040] Entity recognition
[0041] The entity recognition of the present application is divided into two parts: one part is to identify the database tables, columns and values involved in the input SQL; the other part is to identify the entities in the question.
[0042] Suppose the following question and SQL are obtained from Seed Data Pairs as input:
[0043] • Question: "Query the product name and sales amount of the product with the highest sales in 2023."
[0044] • SQL: "SELECT ProductName, SUM(SalesAmount) AS
[0045] TotalSalesAmount
[0046] FROM Sales
[0047] WHERE Year = 2023
[0048] GROUP BY ProductName
[0049] ORDER BY TotalSalesAmount DESC
[0050] LIMIT 1;"
[0051] Then the entity recognition stage outputs are:
[0052] • Table: Sales
[0053] • Fields: ProductName, SalesAmount, Year
[0054] • Entities: SalesAmount, ProductName
[0055] In Step 1, it also includes:
[0056] Step S1: First, use the spaCy library in Python to tokenize and recognize entities in the input natural language question; spaCy is a powerful natural language processing tool that can extract entities such as "sales amount" and "product name" from the question. These entities are usually fields or values involved in database queries, such as "sales amount" may correspond to the SalesAmount field in the database, and "product name" may correspond to the ProductName field.
[0057] Step S2: Extract the involved table and fields from the given SQL query through regular expressions; SQL queries usually contain multiple parts, with the SELECT clause listing the fields to be queried and the FROM clause listing the involved tables. In this step, the invention uses regular expressions to extract the field list and the corresponding table name from the SQL query. For example, in the query SELECT ProductName, SUM(SalesAmount) FROM Sales, the fields ProductName and SalesAmount are extracted, as well as the table Sales.
[0058] In this way, the tables and fields in the natural language question can be identified, and according to the given SQL query and database Schema, the Schema area involved can be accurately located. This provides a basis for subsequent SQL generation and execution verification.
[0059] RAG retrieves corresponding knowledge
[0060] In a Text-to-SQL system, RAG (Retrieval-Augmented Generation) technology is a method to enhance the performance of the generation model by combining an external knowledge base. Through RAG technology, the system can not only rely on database Schema information when generating SQL queries, but also combine business knowledge and context information in the external knowledge base, so as to generate more accurate, reasonable and actual demand NL-SQL question and answer pairs. The following are the specific steps to implement RAG technology:
[0061] Step Y1: First, RAG technology relies on knowledge retrieval, that is, based on the input natural language question and database Schema information, relevant background information is retrieved from the external knowledge base. In the Text-to-SQL system, the external knowledge base usually contains industry knowledge, business rules, term definitions, etc. These knowledge can be metadata of the database, industry standards, document materials, public databases (such as Wikidata), or domain-specific knowledge base. Through retrieval, the RAG system can obtain additional context information related to the current task. For example, if the query is about sales data, the system may retrieve industry knowledge related to "sales" such as "products with the highest sales usually refer to products in the past year";
[0062] Step Y2: Use natural language processing techniques such as BERT to convert the question into an embedding vector, and use vector retrieval techniques (such as FAISS or Elasticsearch) to find similar entries in the knowledge base. This method can capture the semantics of the question, thereby improving the accuracy of retrieval.
[0063] For example, based on the entities identified in the previous stage and the question "Query the product name and sales of the product with the highest sales in 2023.", the knowledge "products with sales higher than 1000 are usually best-selling products" is retrieved from the external knowledge base (such as industry knowledge base). This knowledge can help the model add some possible filtering conditions when generating SQL and ensure that the generated question is business-related.
[0064] SQL-NL data pair synthesis
[0065] The goal of this stage is to generate high-quality SQL statements and corresponding natural language queries to meet the needs of business scenarios and improve the generalization ability of the model. It can be roughly divided into two sub-modules: SQL synthesizer and NL generator, which are responsible for SQL statement generation and natural language query translation respectively, while combining the feedback mechanism of the error memory library to optimize the generation quality.
[0066] The main function of the SQL synthesizer is to generate multiple candidate SQL queries by combining database Schema information and external knowledge. These queries not only contain tables and fields from the Schema, but also integrate relevant information from the external knowledge base. For example, external knowledge may suggest adding "annual total sales" as a filter condition in the SQL query. In this way, the generated SQL queries are no longer limited to simple table field matching, but can intelligently incorporate multi-dimensional information provided by the external knowledge base based on the database Schema, generating more accurate and diverse candidate SQL queries.
[0067] To ensure the diversity of SQL, the invention adopts the method of adjusting the temperature parameter of the reasoning model. Temperature is an important parameter that affects the randomness of query generation. Lower temperature (such as 0.3-0.5) will result in more deterministic and conservative queries, while higher temperature (such as 0.7-1.0) will make the generated queries more diverse and innovative.
[0068] In addition, the system records different types of error data pairs generated during the previous data synthesis process, such as SQL syntax errors, etc. These error data are partially fed back to this stage, and prompts are constructed according to the Few-Shots method, requiring the model to learn from examples in reverse to reduce the probability of repeated errors during generation. This error feedback mechanism effectively improves the robustness and generation quality of the SQL synthesizer.
[0069] The function of the NL generator is relatively simple, mainly responsible for translating the SQL statements synthesized by the SQL synthesizer into corresponding natural language problems. Specifically, the NL generator receives the SQL statements output by the SQL synthesizer and converts them into natural language problems, forming NL pairs with the SQL statements. Similar to the SQL synthesizer, the NL generator also refers to the relevant error data in the error memory library during generation. In this way, the NL generator not only completes the reverse translation of SQL to NL, but also improves the accuracy and naturalness of generated NL under the constraint of error feedback.
[0070] Error feedback mechanism
[0071] The error feedback mechanism designed by the application divides the synthetic data errors into three categories of SQL syntax, execution and NL semantics, and is composed of error detection and feedback.
[0072] In step 4, it also includes:
[0073] Error detection step: In the error detection link, the synthetic data needs to go through three detection processes: syntax detection, execution detection and NL semantic detection. In syntax detection, the SQL statement is parsed with the help of sqlparse and sqlglot libraries, and the try statement block is used to capture syntax abnormalities; execution detection is performed by connecting the database through the sqlite library, and the try statement block is also used to capture runtime errors; to accurately evaluate the semantic consistency of the generated NL-SQL question and answer pair, the application uses the Deepseek-R1 model to score the question and answer pair. It is especially good at complex tasks such as mathematics, code and natural language reasoning, and can be comparable to the OpenAIO1 model on related tasks. Using this model can more accurately determine whether the generated NL-SQL is semantically consistent.
[0074] Error feedback step: In the error feedback stage, different types of error data are fed back to the corresponding link of the synthesis process. SQL syntax errors and SQL execution errors are fed back to the SQL synthesis stage to optimize the generation quality of SQL statements; NL semantic errors are fed back to the NL generation stage to improve the semantic accuracy of natural language queries. Specifically, the application selects the top_k error data with the highest frequency in each type of error in the error memory library and puts it in the Prompt, provides it to the LLM, and clearly indicates the modification suggestions for each type of error, requiring the model to reduce the occurrence of similar errors in this synthesis.
[0075] Through this systematic error detection and feedback mechanism, the application can identify the problems in the synthetic data and use this information to guide the LLM to optimize, thereby continuously improving the quality and reliability of the synthetic data in the Text-to-SQL system.
[0076] The beneficial effects of the application are: the method of the application not only improves the fluency and understandability of natural language questions while ensuring that the generated NL-SQL question and answer pair has a certain logical reasoning complexity, but also ensures the accuracy of the generated SQL statement and its consistency with the question.
[0077] The above content is a further detailed description of the application in combination with a specific preferred embodiment, and cannot be considered as limiting the specific implementation of the application to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, without departing from the concept of the application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the application.
Claims
1. An error feedback-based NL2SQL large model training data synthesis method, characterized in that, The method comprises the following steps: Step 1: identify entities in the seed question-answer pair, including Schema areas in the database and entities in natural language; Step 2: match the knowledge related to the question and entity in the knowledge base using the RAG technology; Step 3: generate the corresponding SQL query according to the obtained knowledge and entity information, and convert it into a natural language question; Step 4: quality evaluation of the generated SQL question-answer pair to ensure that high-quality NL-SQL question-answer pairs are added to the training set, and NL-SQL question-answer pairs with errors are fed back to the NL-SQL question-answer generation link to avoid the same errors from occurring repeatedly.
2. The NL2SQL large model training data synthesis method according to claim 1, characterized in that, In step 1, the entity recognition is divided into two parts: one part is to identify the database tables, columns and values involved in the input SQL; the other part is to identify the entities in the question.
3. The NL2SQL large model training data synthesis method according to claim 2, characterized in that, In step 1, it also includes: Step S1: use the spaCy library in Python to perform word segmentation and entity recognition on the input natural language question; Step S2: extract the tables and fields involved from the given SQL query through regular expressions.
4. The NL2SQL large model training data synthesis method according to claim 1, characterized in that, In step 2, it also includes: Step Y1: based on the input natural language question and database Schema information, retrieve the background information related to the task from the external knowledge base; Step Y2: use natural language processing technology to convert the question into an embedding vector, and find similar items in the knowledge base through vector retrieval technology. 5.The NL2SQL large model training data synthesis method according to claim 1, characterized in that, In step 3, the SQL synthesizer and NL generator are responsible for SQL statement generation and natural language query translation, while the feedback mechanism of the error memory library is used to optimize the generation quality. 6.The NL2SQL large model training data synthesis method according to claim 5, characterized in that, The SQL synthesizer is responsible for combining database Schema information and external knowledge to generate candidate SQL queries, which not only contain tables and fields from the Schema, but also integrate relevant information from the external knowledge base; The NL generator is responsible for receiving the SQL statements output by the SQL synthesizer and converting them into natural language questions, forming NL pairs with the SQL statements.
7. The NL2SQL large model training data synthesis method according to claim 5, characterized in that, In step 3, the temperature parameter method of adjusting the reasoning model is adopted to ensure the diversity of SQL query generation. 8.The NL2SQL large model training data synthesis method according to claim 1, characterized in that, In step 4, it also includes: Error detection step: in the error detection link, the synthesized data needs to go through three detection processes: syntax detection, execution detection and NL semantic detection; In syntax detection, sqlparse and sqlglot libraries are used to parse SQL statements, and try statement blocks are used to capture syntax abnormalities; Execution detection is performed by connecting the database through the sqlite library, and try statement blocks are also used to capture runtime errors; NL semantic detection uses a large language model to score the question-answer pair; Error feedback step: in the error feedback stage, different types of error data are fed back to the corresponding link of the synthesis process; SQL syntax errors and SQL execution errors are fed back to the SQL synthesis stage to optimize the generation quality of SQL statements; NL semantic errors are fed back to the NL generation stage to improve the semantic accuracy of natural language queries.
9. A system for error feedback based NL2SQL large model training data synthesis, characterized in that, It includes: A memory, a processor, and a computer program stored on the memory, the computer program configured to implement the steps of the method of any one of claims 1-8 when invoked by the processor.
10. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, the computer program configured to implement the steps of the method of any one of claims 1-8 when invoked by a processor.