Method and device for generating text2sql training corpus based on bidirectional enhancement and multi-order supervision
Through two-way enhancement and multi-order supervision methods, high-quality text2sql training corpus is automatically generated, solving the problem of existing methods relying on manual annotation and application scenario limitations, and improving the adaptability and generalization capabilities of the model.
Patent Information
- Application Number
- CN202411870948.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-18
- Publication Date
- 2025-05-06
AI Technical Summary
The existing text2sql training corpus generation method relies on manual annotation and is limited to specific application scenarios, resulting in inefficiency, high cost and poor model adaptability.
Using two-way enhancement and multi-order supervision methods, through the assistance of multi-stage supervision and review mechanisms and large language models, a large number of reliable and general "problem-SQL" pairs are automatically generated to improve the efficiency and quality of corpus generation.
It significantly reduces the cost of manual labeling, enhances the diversity and reliability of the corpus, and improves the adaptability and generalization capabilities of the model.
Smart Images
Figure CN119940527A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and more specifically, to a method and device for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision. Background Art
[0002] In the field of natural language processing, the task of converting natural language queries (NL questions) into structured query language (SQL) (text2sql) is of great significance for realizing intelligent question-answering systems and database automation operations.
[0003] Existing methods have significant defects in generating training corpus: first, they are highly dependent on manual annotation and checking, which leads to low efficiency and high cost; second, the training corpus is often limited to specific application scenarios and lacks diversity, which limits the model's ability to adapt to complex and changing environments. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and device for generating text2sql training corpus based on bidirectional enhancement and multi-stage supervision. By combining bidirectional data enhancement technology and a multi-stage supervision review mechanism, a large number of reliable and universal "question-SQL" pairs are automatically generated with a relatively low annotation cost, which significantly improves the efficiency and quality of corpus generation, thereby improving the adaptability and generalization ability of the text2sql model.
[0005] The object of the present invention is achieved through the following solutions:
[0006] A method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision includes the following steps:
[0007] S1, question-to-SQL enhancement, specifically includes: S11, collecting natural language questions from users and annotating corresponding SQL statements to form a seed set; S12, performing multi-stage supervision and review enhancement;
[0008] S2, SQL to question enhancement, specifically includes: S21, using SQL templates, combined with database table structure and fields, to generate general "question-SQL" pairs; S22, for entities in the library table, enumerate their possible values, and reverse substitute them into the question template to generate diversified questions; S23, using a large language model to restate the generated questions in natural language to ensure that the questions conform to Chinese grammatical habits while keeping the original meaning unchanged.
[0009] Furthermore, in step S11, the natural language questions collected from the users are real natural language questions.
[0010] Furthermore, in step S11, the annotation is manual annotation.
[0011] Furthermore, in step S12, the multi-stage supervisory review enhancement is performed, specifically including the following sub-steps:
[0012] Phase 1: Combine the seed set and database table description and use the large model to expand the problem set;
[0013] Phase 2: The large model plays a supervisory role, reviews the quality of questions, filters out low-quality and unanswerable questions, and corrects unclear questions.
[0014] Phase 3: Based on the code generation capability of the large model, combined with the "question-SQL" pairs in the seed set, SQL statements are generated for the expanded questions;
[0015] Phase 4: Using the SQL syntax knowledge and database table design information of the big model, the generated “question-SQL” pairs are reviewed for correctness to ensure the accuracy of the corpus.
[0016] Further, in step S21, the SQL template includes the SQL template in the data set Spider.
[0017] Furthermore, in the first stage, the seed set and the database table description are combined to expand the question set using the large model, which specifically includes the sub-steps of utilizing the contextual understanding capability of the large language model, taking the seed set as a positive example, and combining the database table structure and field information to generate a diverse question set.
[0018] Further, in the first stage, the large model includes a large language model qwen2-7b.
[0019] Further, in the second stage, the large model includes a large language model qwen2-7b.
[0020] Further, in the third stage, the large model includes a large language model qwen2-7b-coder.
[0021] A text2sql training corpus generation device based on bidirectional enhancement and multi-level supervision comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, any of the above methods is executed.
[0022] The beneficial effects of the present invention include:
[0023] (1) The present invention can reduce the cost of manual annotation: through automatic generation and multi-stage supervision and review, it significantly reduces manual participation and reduces the cost of corpus generation.
[0024] (2) The present invention can enhance the diversity of corpus: the proposed bidirectional enhancement strategy ensures that the training corpus covers a wider range of database scenarios and improves the generalization ability of the model.
[0025] (3) The present invention can improve the quality of corpus: the multi-stage supervision and review mechanism effectively filters out low-quality and erroneous corpus, ensuring the high reliability of the training corpus. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0027] Figure 1 Enhanced flowchart for Question to SQL;
[0028] Figure 2 Enhanced flowchart for SQL to problem. DETAILED DESCRIPTION
[0029] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.
[0030] In view of the problems in the background, the inventors of this application believe that the existing methods have the following technical problems: 1) High cost of manual annotation: A large amount of training corpus requires manual annotation and verification, which is not only time-consuming and labor-intensive, but also increases the economic burden of the project. 2) Limited application scenarios: The database scenarios covered by the training corpus are limited, making it difficult to cope with diverse practical needs, resulting in poor performance of the model in applications in unknown or new fields.
[0031] In order to solve the above technical problems, the solution of the present invention combines bidirectional data enhancement technology and multi-stage supervision and review mechanism to automatically generate a large number of reliable and universal "question-SQL" pairs with a relatively low annotation cost, significantly improving the efficiency and quality of corpus generation. Specific technical solution implementation: 1) Enhancement from question to SQL. First, collect real natural language questions from users, and manually annotate the corresponding SQL statements to form a high-quality seed set. Then perform multi-stage supervision and review enhancement. Phase 1: Combine the seed set and database table description, and use the big model to expand the question set. Phase 2: The big model plays a supervisory role, conducts question quality review, filters low-quality and unanswerable questions, and corrects unclear questions. Phase 3: Based on the code generation capability of the big model, combined with the "question-SQL" pairs in the seed set, automatically generate SQL statements for the expanded questions. Phase 4: Use the SQL syntax knowledge of the big model and the database table design information to conduct a correctness review of the generated "question-SQL" pairs to ensure the accuracy of the corpus. 2) Enhancement from SQL to question. First, use the common SQL templates in the public data set, combined with the database table structure and fields, to generate a universal "question-SQL" pair. Then, for the special entities in the database table, we enumerate their possible values and substitute them into the question template to generate a variety of questions. Finally, we use the large language model to restate the generated questions in natural language to ensure that the questions conform to Chinese grammatical habits while keeping the original meaning unchanged.
[0032] The above embodiments mainly provide an overview of the core content of the present invention. The specific implementation details, experimental verification results and further optimization solutions will be described in detail in the subsequent embodiments. The embodiments of the present invention aim to provide an efficient and low-cost method for generating text2sql training corpus to promote the wide application of natural language processing technology in the field of database query automation. In other embodiments, the enhanced process from question to SQL and the enhanced process from SQL to question are specifically included.
[0033] like Figure 1 As shown, the enhanced process from question to SQL is as follows:
[0034] Step 1: Seed set construction
[0035] First, we collect real natural language questions from users and manually annotate the corresponding SQL statements to form a high-quality seed set. For example, the seed set can contain the following "question-SQL" pairs:
[0036] Problem 1: Get the username and age of all users.
[0037] SQL:SELECT username,age FROM Users;
[0038] Question 2: Get the product name, order date, and age of the ordering user for all orders. SQL:SELECT Orders.product_name, Orders.order_date, Users.age FROM Orders JOIN Users ON Orders.user_id = Users.user_id;
[0039] Question 3: Get all orders with product name "Laptop" and their prices.
[0040] SQL: SELECT Orders.order_id,Products.price FROM Orders JOIN ProductsON Orders.product_id=Products.product_id WHERE Products.product_name='Laptop';
[0041] These seed sets will serve as the basis for subsequent generation and review.
[0042] Step 2: Enhanced multi-stage supervisory review
[0043] Step 2.1, Phase 1: Diversification Question Expansion
[0044] By using the contextual understanding ability of the large language model qwen2-7b, combined with the database table structure and field information (such as Table 1: Users, including user_id, user_name, age, email_addr fields; Table 2: Orders, including order_id, product_id, order_date, user_id fields; Table 3: Products, including product_id, product_name, price fields), a diverse set of questions can be generated. For example, by inputting "question expansion prompt words" and database table design information, the large model can generate questions like "list the usernames and email addresses of all users".
[0045] The prompt for question expansion is: You are a text2sql expert. Your task is to refer to the "Question Seed Set" and combine it with the "Database Table Design Information". Please think step by step according to the following steps: 1. Enrich the question angles from the perspectives of when, where, which, where, how, why, etc.; 2. Carefully read the database table design and give possible questions from the perspectives of single table and multi-table joint query 3. Simulate the questioning methods or habits in the seed list to make the questions more in line with Chinese expression habits. Raise as many possible user questions as possible from multiple dimensions. Return format requirements: It is required to return as a string list, and each element is a question string. The question seed set is: xxx. The database table design information is: xxxx.
[0046] Step 2.2, Phase 2: Question Quality Review
[0047] The large model qwen2-72b is used as a supervisor to review the quality of the generated questions. By inputting the "question quality generation prompt words" and the generated question set, the large model will filter out low-quality and unanswerable questions and correct unclear questions. For example, "analyze the user's personality information from the purchase record" is a question that cannot be answered based on the database table content. The question "list the age and address of the user in the user table whose username is Zhang San" can be corrected to "list Zhang San's age and email address."
[0048] The prompt for question quality generation is: You are a question quality review expert. Your task is to refer to the "database table design information" and conduct a quality review of the questions in the "question list". Please think step by step according to the following steps: 1. Questions that cannot be answered by the existing database are marked as filtered; 2. Question expressions that do not conform to Chinese habits are marked as corrected, and the corrected questions are given; 3. Questions that have no meaning are marked as filtered. Return format requirements: It is required to return in json format, with the key being the question number, question type, and rewritten question. The question list is: xxx. The database table design information is: xxxx.
[0049] Step 2.3, Phase 3: Automatically generate SQL statements
[0050] Based on the code generation capability of the big model qwen2-7b-coder, combined with the "question-SQL" pairs in the seed set, SQL statements are automatically generated for each question in the question list. By entering the prompt word "automatically generate SQL prompt word" and the generated question, the big model will generate the corresponding SQL statement. For example, for the question "list the username and email address of all users", the big model will generate SELECT username, email FROM Users; (assuming that the Users table contains the email field).
[0051] The automatically generated SQL prompt is: You are a text2sql expert. Your task is to refer to the "Question Seed Set" and combine it with the "Database Table Design Information" to give the SQL statement corresponding to each question in the "Question List". Return format requirements: It is required to return in json format, with the key being the question number and the corresponding SQL statement. The question seed set is: xxx. The database table design information is: xxxx. The question list is: xxx.
[0052] Step 2.4, Phase 4: Correctness review of “question-SQL pair”
[0053] Using the SQL grammar knowledge and database table design information of the big model qwen2-72b, the generated "question-SQL" pairs are reviewed for correctness. By entering the prompt word "question-SQL pair correctness review prompt word", the generated questions and SQL statements, the big model will ensure the accuracy of the corpus. For example, for the question "list the usernames and email addresses of all users" and the generated SQL statement SELECT username,email FROM Users, the big model will verify its correctness.
[0054] The prompt for the correctness review of the problem-SQL pair is: You are a text2sql expert. Your task is to refer to the "database table design information" and conduct a quality review of the "problem-SQL pair" list. Please think step by step according to the following steps: 1. If the SQL statement is correct, skip it; 2. If the SQL statement is wrong, mark it as an error and give the corrected SQL; Return format requirements: It is required to return in json format, with the key being the "problem-SQL pair" serial number and the rewritten SQL. The "problem-SQL pair" list is: xxx. The database table design information is: xxxx.
[0055] Step 3, SQL to problem enhancement
[0056] Step 3.1: Collecting common SQL templates
[0057] like Figure 2 As shown in the figure, common SQL templates in the public dataset Spider are used, combined with database table structure and fields, to generate common "question-SQL" pairs. For example, for the SQL template "SELECT {COLUMN} FROM {TABLE} GROUP BY {COLUMN} ORDER BY COUNT(*) ASC LIMIT 1", the question "return the lowest {COLUMN} of {TABLE}" can be generated.
[0058] Step 3.2, special entity enumeration
[0059] For special entities in the database table (such as name, type, etc.), enumerate their possible values and substitute them into the question template to generate diversified questions. For example, for the username field in the Users table, you can enumerate its possible values (such as "Alice", "Bob", etc.) and generate the question "Return the age of the user named Alice".
[0060] Step 3.3, large model recap
[0061] Use the large language model qwen2-72b (or higher-level models) to restate the generated question in natural language, ensuring that the question conforms to Chinese grammatical conventions while keeping the original meaning unchanged. By entering the prompt word "large model restate prompt word" and the generated question, the large model will restate the question. For example, "return the age field of users whose username is equal to Alice" is restated as "query Alice's age information."
[0062] Rephrase prompt for large model: You are a text2sql expert. Your task is to read each question in the "question list" carefully and rephrase it in a way that conforms to Chinese expression habits without changing the original meaning. Return format requirements: It is required to return in json format, with the key being the question number and the rewritten question. The question list is: xxx.
[0063] Through the above-mentioned bidirectional enhancement and multi-level supervision methods, the present invention can automatically generate a large number of reliable and universal "question-SQL" pairs, significantly improving the efficiency and quality of corpus generation, reducing the cost of manual annotation, and enhancing the diversity and reliability of the corpus.
[0064] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.
[0065] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.
[0066] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.
Claims
1. A method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision, characterized in that: The steps include: S1, question-to-SQL enhancement, specifically includes: S11, collecting natural language questions from users and annotating corresponding SQL statements to form a seed set; S12, performing multi-stage supervision and review enhancement; S2, SQL to question enhancement, specifically includes: S21, using SQL templates, combined with database table structure and fields, to generate general "question-SQL" pairs; S22, for entities in the library table, enumerate their possible values, and reverse substitute them into the question template to generate diversified questions; S23, using a large language model to restate the generated questions in natural language to ensure that the questions conform to Chinese grammatical habits while keeping the original meaning unchanged.
2. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 1, characterized in that: In step S11, the natural language questions collected from the user are real natural language questions.
3. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 1, characterized in that: In step S11, the annotation is manual annotation.
4. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 1, characterized in that: In step S12, the multi-stage supervisory review enhancement is performed, specifically including the following sub-steps: Phase 1: Combine the seed set and database table description and use the large model to expand the problem set; Phase 2: The large model plays a supervisory role, reviews the quality of questions, filters out low-quality and unanswerable questions, and corrects unclear questions. Phase 3: Based on the code generation capability of the large model, combined with the "question-SQL" pairs in the seed set, SQL statements are generated for the expanded questions; Phase 4: Using the SQL syntax knowledge and database table design information of the big model, the generated "question-SQL" pairs are reviewed for correctness to ensure the accuracy of the corpus.
5. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 1, characterized in that: In step S21, the SQL template includes the SQL template in the dataset Spider.
6. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 4, characterized in that: In the first stage, the seed set and the database table description are combined to expand the question set using the large model, which specifically includes the following sub-steps: using the context understanding ability of the large language model, taking the seed set as a positive example, and combining the database table structure and field information to generate a diverse question set.
7. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 4, characterized in that: In the first stage, the large model includes the large language model qwen2-7b.
8. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 7, characterized in that: In the second stage, the large model includes a large language model qwen2-7b.
9. The method for generating text2sql training corpus based on bidirectional enhancement and multi-level supervision according to claim 4, characterized in that: In the third stage, the large model includes a large language model qwen2-7b-coder.
10. A text2sql training corpus generation device based on bidirectional enhancement and multi-level supervision, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is loaded by the processor, the method according to any one of claims 1 to 9 is executed.