Method and system for constructing NL2SQL data set in large model scene

Through the NL2SQL dataset construction method involving business personnel, combined with the problem of large-model expansion and difficulty level annotation, the problems of inefficient dataset construction and insufficient generalization capabilities in the existing technology are solved, and high-quality and low-cost datasets are achieved Construction improves the generalization ability of the model and the accuracy of SQL generation.

CN119003570APending Publication Date: 2024-11-22SHANDONG LANGCHAO YUNTOU INFORMATION TECH CO LTD

Patent Information

Application Number
CN202411002579.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Building high-quality NL2SQL datasets is inefficient and costly, making it difficult to cover all possible query scenarios and expression methods, limiting the generalization ability of the model.

Method used

Through business personnel participating in problem expression and SQL statement verification, combined with large-scale model expansion problems, generalized problem and template problem data sets are constructed, and difficulty level annotations are performed, merged into training sets and test sets, and used to supervise fine-tuning of large models.

Benefits of technology

It improves the quality and business relevance of the data set, increases diversity and novelty, improves the generalization ability of the model and the correctness and compliance of SQL generation, and reduces the cost of model training.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003570A_ABST
    Figure CN119003570A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of large models, in particular to a method and system for constructing an NL2SQL data set in a large model scene, and the method comprises the following steps: data set construction: constructing a generalization problem data set and a template problem data set according to business requirements and user function points; cleaning and difficulty grade labeling are carried out on the generalization problem and the template problem; merging the generalization problem and the template problem, and extracting a training set and a test set in proportion; utilizing the training set and the test set to supervise and finely adjust the large model so as to improve the ability of the large model to generate an SQL according to a natural language; the method and the system for constructing the NL2SQL dataset in the large model scene have the beneficial effects that the quality and business correlation of the dataset are ensured through the provision of the business personnel participating in the problem expression and the verification of the SQL statement. And meanwhile, the problem is expanded by utilizing a large model, so that the diversity and novelty of a data set are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of large models, and in particular to a method and system for constructing an NL2SQL data set in a large model scenario. Background Art

[0002] In the field of Natural Language Processing (NLP), the task of converting Natural Language (NL) to Structured Query Language (SQL), namely NL2SQL, is an important research direction. This technology aims to enable non-technical users to query databases through natural language without writing complex SQL statements.

[0003] However, building high-quality NL2SQL datasets is a challenge in this field, as the datasets need to cover a wide range of query scenarios and have sufficient diversity and accuracy to effectively train and evaluate NL2SQL models.

[0004] Traditional NL2SQL dataset construction methods often rely on manually writing questions and their corresponding SQL statements. Although this method is accurate, it is inefficient and costly. In addition, manually constructed datasets often fail to cover all possible query scenarios and expressions, limiting the generalization ability of the model. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for constructing an NL2SQL data set in a large model scenario to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for constructing an NL2SQL data set in a large model scenario, the method comprising the following steps:

[0007] Dataset construction: construct two data sets, generalization questions and template questions, based on business needs and user function points;

[0008] Clean and label the difficulty level of generalization questions and template questions;

[0009] Combine the generalization problem and the template problem, and extract the training set and test set in proportion;

[0010] The training set and test set are used to perform supervised fine-tuning on the large model to improve the large model's ability to generate SQL based on natural language.

[0011] Preferably, the steps of constructing a generalization problem dataset include:

[0012] Business personnel provide problem statements, input relevant problems, table structure information and some data into the big model, and generate corresponding SQL statements;

[0013] Business personnel check the SQL results based on the business database and manually mark and modify the incorrect SQL to ensure its correctness and compliance;

[0014] Use the big model to expand the problem, change the expression method but not the meaning, and get the expanded problem;

[0015] Verify the expanded questions to ensure that the expanded semantics are consistent with the original semantics;

[0016] Provide the expanded questions, table structure information and relevant business background knowledge to the big model to generate the corresponding SQL statements;

[0017] Compare the SQL execution results generated by the expanded question with the SQL execution results of the seed question, retain the questions and SQL with consistent results, and obtain the initial generalized question data.

[0018] Preferably, the steps of constructing the template question data set include:

[0019] Business personnel provide problem descriptions in real scenarios, including multiple categories and dimensions;

[0020] Input relevant questions, table structure information and some data into the big model, design and build prompts, and call the big model to automatically generate SQL statements that conform to syntax and semantics;

[0021] Execute SQL, check and correct the incorrect SQL statements, and obtain the initial template problem and SQL statements;

[0022] Instantiate the initial template problem, change it into a template form, and generate new sample data;

[0023] After the splicing is completed, it is input into the database to execute the query, retain the correct questions and SQL data sets, and obtain the verified standardized template question data set.

[0024] Preferably, before constructing the data set, an initial seed data set is generated using the problem statement, table structure information and partial data provided by the business personnel.

[0025] Preferably, the questions in the initial seed data set are expanded using a large model to obtain expanded questions; the expanded questions are verified to ensure that the expanded semantics are consistent with the original semantics; and the SQL statements corresponding to the expanded questions are generated using the large model to form initial generalized question data.

[0026] A NL2SQL dataset system is constructed in a large model scenario, the system comprising a dataset construction module, a dataset preprocessing module, a dataset classification module and a dataset adjustment module;

[0027] The data set construction module constructs two data sets: generalization questions and template questions according to business requirements and user function points;

[0028] The dataset preprocessing module cleans and labels the difficulty levels of generalization questions and template questions;

[0029] The dataset classification module combines the generalization problem and the template problem, and extracts the training set and the test set in proportion;

[0030] The dataset adjustment module uses the training set and the test set to perform supervised fine-tuning on the large model to improve the large model's ability to generate SQL based on natural language.

[0031] Preferably, the step of constructing a generalized problem dataset in the dataset construction module includes:

[0032] Business personnel provide problem statements, input relevant problems, table structure information and some data into the big model, and generate corresponding SQL statements;

[0033] Business personnel check the SQL results based on the business database and manually mark and modify the incorrect SQL to ensure its correctness and compliance;

[0034] Use the big model to expand the problem, change the expression method but not the meaning, and get the expanded problem;

[0035] Verify the expanded questions to ensure that the expanded semantics are consistent with the original semantics;

[0036] Provide the expanded questions, table structure information and relevant business background knowledge to the big model to generate the corresponding SQL statements;

[0037] Compare the SQL execution results generated by the expanded question with the SQL execution results of the seed question, retain the questions and SQL with consistent results, and obtain the initial generalized question data.

[0038] Preferably, the step of constructing the template question dataset in the dataset construction module includes:

[0039] Business personnel provide problem descriptions in real scenarios, including multiple categories and dimensions;

[0040] Input relevant questions, table structure information and some data into the big model, design and build prompts, and call the big model to automatically generate SQL statements that conform to syntax and semantics;

[0041] Execute SQL, check and correct the incorrect SQL statements, and obtain the initial template problem and SQL statements;

[0042] Instantiate the initial template problem, change it into a template form, and generate new sample data;

[0043] After the splicing is completed, it is input into the database to execute the query, retain the correct questions and SQL data sets, and obtain the verified standardized template question data set.

[0044] Preferably, the dataset construction module generates an initial seed dataset using the problem statement, table structure information and partial data provided by the business personnel before the dataset is constructed.

[0045] Preferably, the data set construction module uses a large model to expand the questions in the initial seed data set to obtain expanded questions; verifies the expanded questions to ensure that the expanded semantics are consistent with the original semantics; and uses the large model to generate SQL statements corresponding to the expanded questions to form initial generalized question data.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The method and system for constructing NL2SQL data sets in a large model scenario proposed in the present invention ensure the quality and business relevance of the data sets by allowing business personnel to participate in providing problem statements and verifying SQL statements. At the same time, the large model is used to expand the problems, increasing the diversity and novelty of the data sets.

[0048] By constructing two data sets, generalized questions and template questions, and labeling them according to their difficulty levels, the trained large model can better adapt to questions of different difficulty levels and types, improving the generalization ability of the model and achieving the training of the model to generate SQL of a specific difficulty level.

[0049] By generating SQL statements that conform to syntax and semantics and manually checking and correcting them, the correctness and compliance of the generated SQL statements are ensured, thereby improving the user experience when using large models for database queries.

[0050] By constructing a target data set that meets the format requirements and extracting generalization problems and template problems according to the required difficulty level and proportion, the trained large model can converge faster, reducing the cost of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose and technical solution of the present invention clearly and completely described, and the advantages more clearly understood, the embodiments of the present invention are further described in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments, and are only used to explain the embodiments of the present invention, and are not used to limit the embodiments of the present invention. All other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1

[0054] See also Figure 1 The present invention provides a technical solution: a method for constructing an NL2SQL data set in a large model scenario, the method comprising the following steps:

[0055] Construct an sft dataset to improve the ability of the large model to generate SQL based on natural language in specific scenarios. The target dataset includes table meta-information files (including table names and field names with Chinese annotations) and structured files (including Chinese query statements, correct SQL statements corresponding to Chinese query statements, SQL execution results, table name information involved in SQL, and difficulty level annotations of SQL statements)

[0056] When constructing the target dataset, we first construct the generalization problem and the template problem separately, then merge them after unified cleaning and difficulty level labeling, and extract the training set and test set in proportion for supervised fine-tuning of the large model.

[0057] Dataset construction strategy:

[0058] 1. Generalization problem natural statement and SQL statement construction process:

[0059] 1) Relevant business personnel provide problem statements based on the business user's required functional points and usage. The relevant questions, table structure information, and part of the data in the table are input into the big model, and the big model with an open source commercial agreement generates the corresponding SQL statement. The business personnel then execute the SQL check results in combination with the business database, and manually mark and modify the incorrect SQL to ensure its correctness and compliance, and obtain the initial seed data set.

[0060] 2) Different questions are expanded by a large model with an open source commercial agreement, changing the expression method but not the meaning. The new questions are fine-tuned on the basis of maintaining the original meaning, so that they are similar to the original questions but different in meaning, ensuring that they increase diversity and novelty while maintaining the original meaning. The number of new questions expanded from an original question is determined by the size of the target data set to be generated. The expanded questions are manually verified to ensure that the expanded semantics are consistent with the original semantics.

[0061] 3) Provide the problem, table structure information (table fields, type and meaning of each field, primary key and foreign key relationship), relevant business background knowledge including terminology explanations in business scenarios and specification requirements for generating SQL statements to a large model with an open source commercial agreement to generate corresponding SQL statements.

[0062] 4) Compare the SQL execution results generated by the expanded question with the SQL execution results of the seed question, and keep the questions and SQL with the same results. One seed question corresponds to multiple expanded questions and multiple candidate correct SQLs. That is, generalized question data is obtained. When constructing the data set, each question can be randomly selected from the candidate SQLs of the corresponding SQL group.

[0063] 2. Template question natural statement and SQL statement construction process:

[0064] 1) Relevant business personnel provide problem descriptions in actual scenarios based on business needs. Problems include multiple categories (details, trends, distributions, totals) and multiple dimensions (historical problems, real-time problems). Relevant problems, table structure information, and part of the data in the table are input into the big model, prompts are designed and constructed, and the big model with open source commercial protocols is called to automatically generate SQL statements that conform to syntax and semantics.

[0065] 2) Execute SQL. If errors or non-compliant SQL statements are found, the business personnel will check and correct the incorrect SQL in combination with the business database to obtain the initial template problem and SQL statement. Manual inspection checks the SQL generated by the large model one by one according to certain rules to ensure that the syntax is correct, the semantics are clear, and it meets business needs. If errors or questions are found, make adjustments and corrections in a timely manner. Check the fields involved in the SQL statement to ensure that it is consistent with the database table structure and meets business needs. If errors or inconsistencies are found,

[0066] Make adjustments and corrections from time to time.

[0067] 3) Instantiate the original template problem and change it into a template form. Write a script to randomly extract information to fill in and generate new samples. According to the example problem and SQL template, perform instance splicing on SQL, such as random replacement of time, place, and category, or change of sorting and condition comparison, to generate more new sample data.

[0068] 4) After the splicing is completed, enter it into the database to execute the query and retain the correct questions and SQL

[0069] Dataset,

[0070] Get the verified standardized template question dataset.

[0071] 3. Target dataset construction process:

[0072] 1) Perform cleaning to remove meaningless characters.

[0073] 2) The script is used to mark the difficulty level. The more restrictions and aggregate functions a SQL statement contains, the deeper the prediction SQL statement is. The script will count the number of query conditions of the where function, the number of restrictions of the groupby function, and the number of tables involved in multi-table join query, and then mark the difficulty level according to the corresponding rules.

[0074] 3) According to the required difficulty level and proportion, generalized questions and template questions are extracted to construct a target data set that meets the format requirements. It includes a prompt text with a predetermined format and an answer text corresponding to the prompt text, wherein the prompt text includes a Chinese query statement, the meta information of the database table queried by the Chinese query statement, and the Chinese annotations to the meta information; the answer text includes the SQL statement corresponding to the Chinese query statement.

[0075] Quality Control Mechanism:

[0076] 1. Continuous monitoring and updating: Establish a continuous monitoring mechanism for data set quality, regularly evaluate data set performance, and update it based on business changes and model performance feedback.

[0077] 2. User feedback loop: Introduce a user feedback loop mechanism to allow actual users to annotate and evaluate the dataset during use, further improving the quality of the dataset.

[0078] Adjustment and optimization mechanism:

[0079] Flexible dataset configuration: Flexible dataset configuration options are provided based on business needs, such as customizing datasets in specific fields and adjusting the proportion of data at different difficulty levels.

[0080] Multi-stage training strategy: In the process of large model training, a multi-stage training strategy is introduced. First, pre-train on simpler data, and then gradually increase the complexity and difficulty of the data to improve the model training efficiency and final performance.

[0081] Embodiment 2

[0082] Based on the first embodiment, a system for constructing NL2SQL datasets in a large model scenario is proposed, wherein the system is composed of a dataset construction module, a dataset preprocessing module, a dataset classification module, and a dataset adjustment module;

[0083] The data set construction module constructs two data sets, generalized questions and template questions, according to business requirements and user function points. Before the data set is constructed, the initial seed data set is generated using the problem statement, table structure information and part of the data provided by the business personnel. The questions in the initial seed data set are expanded using the big model to obtain the expanded questions. The expanded questions are verified to ensure that the expanded semantics are consistent with the original semantics. The SQL statements corresponding to the expanded questions are generated using the big model to form the initial generalized question data. The steps of constructing the generalized question data set include:

[0084] Business personnel provide problem statements, input relevant problems, table structure information and some data into the big model, and generate corresponding SQL statements;

[0085] Business personnel check the SQL results based on the business database and manually mark and modify the incorrect SQL to ensure its correctness and compliance;

[0086] Use the big model to expand the problem, change the expression method but not the meaning, and get the expanded problem;

[0087] Verify the expanded questions to ensure that the expanded semantics are consistent with the original semantics;

[0088] Provide the expanded questions, table structure information and relevant business background knowledge to the big model to generate the corresponding SQL statements;

[0089] Compare the SQL execution results generated by the expanded question with the SQL execution results of the seed question, retain the questions and SQL with consistent results, and obtain the initial generalized question data.

[0090] The steps to construct the template question dataset include:

[0091] Business personnel provide problem descriptions in real scenarios, including multiple categories and dimensions;

[0092] Input relevant questions, table structure information and some data into the big model, design and build prompts, and call the big model to automatically generate SQL statements that conform to syntax and semantics;

[0093] Execute SQL, check and correct the incorrect SQL statements, and obtain the initial template problem and SQL statements;

[0094] Instantiate the initial template problem, change it into a template form, and generate new sample data;

[0095] After the splicing is completed, it is input into the database to execute the query, retain the correct questions and SQL data sets, and obtain the verified standardized template question data set.

[0096] The dataset preprocessing module cleans and labels the difficulty levels of generalization questions and template questions;

[0097] The dataset classification module combines the generalization problem and the template problem, and extracts the training set and the test set in proportion;

[0098] The dataset adjustment module uses the training set and the test set to perform supervised fine-tuning on the large model to improve the large model's ability to generate SQL based on natural language.

[0099] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for constructing an NL2SQL dataset in a large model scenario, characterized by: The method comprises the following steps: Dataset construction: Based on business requirements and user function points, two data sets are constructed: generalization questions and template questions. Clean and label the difficulty level of generalization questions and template questions; Combine the generalization problem and the template problem, and extract the training set and test set in proportion; The training set and test set are used to perform supervised fine-tuning on the large model to improve the large model's ability to generate SQL based on natural language.

2. The method for constructing an NL2SQL data set in a large model scenario according to claim 1, characterized in that: The steps to construct a generalization problem dataset include: Business personnel provide problem statements, input relevant problems, table structure information and some data into the big model, and generate corresponding SQL statements; Business personnel check the SQL results based on the business database and manually mark and modify the incorrect SQL to ensure its correctness and compliance; Use the big model to expand the problem, change the expression method but not the meaning, and get the expanded problem; Verify the expanded questions to ensure that the expanded semantics are consistent with the original semantics; Provide the expanded questions, table structure information and relevant business background knowledge to the big model to generate the corresponding SQL statements; Compare the SQL execution results generated by the expanded question with the SQL execution results of the seed question, retain the questions and SQL with consistent results, and obtain the initial generalized question data.

3. The method for constructing an NL2SQL data set in a large model scenario according to claim 1, characterized in that: The steps to construct the template question dataset include: Business personnel provide problem descriptions in real scenarios, including multiple categories and dimensions; Input relevant questions, table structure information and some data into the big model, design and build prompts, and call the big model to automatically generate SQL statements that conform to syntax and semantics; Execute SQL, check and correct the incorrect SQL statements, and obtain the initial template problem and SQL statements; Instantiate the initial template problem, change it into a template form, and generate new sample data; After the splicing is completed, it is input into the database to execute the query, retain the correct questions and SQL data sets, and obtain the verified standardized template question data set.

4. The method for constructing an NL2SQL data set in a large model scenario according to claim 1, characterized in that: Before constructing the dataset, the initial seed dataset is generated using the problem statement, table structure information, and partial data provided by business personnel.

5. The method for constructing an NL2SQL data set in a large model scenario according to claim 4, characterized in that: Use the big model to expand the questions in the initial seed data set to obtain the expanded questions; verify the expanded questions to ensure that the expanded semantics are consistent with the original semantics; use the big model to generate SQL statements corresponding to the expanded questions to form the initial generalized question data.

6. A system for constructing an NL2SQL dataset in a large model scenario according to the method for constructing an NL2SQL dataset in a large model scenario according to any one of claims 1 to 5, characterized in that: The system consists of a data set construction module, a data set preprocessing module, a data set classification module and a data set adjustment module; The data set construction module constructs two data sets: generalization questions and template questions according to business requirements and user function points; The dataset preprocessing module cleans and labels the difficulty levels of generalization questions and template questions; The dataset classification module combines the generalization problem and the template problem, and extracts the training set and the test set in proportion; The dataset adjustment module uses the training set and the test set to perform supervised fine-tuning on the large model to improve the large model's ability to generate SQL based on natural language.

7. The system for constructing NL2SQL data sets in a large model scenario according to claim 6, characterized in that: The steps of constructing a generalization problem dataset in the dataset construction module include: Business personnel provide problem statements, input relevant problems, table structure information and some data into the big model, and generate corresponding SQL statements; Business personnel check the SQL results based on the business database and manually mark and modify the incorrect SQL to ensure its correctness and compliance; Use the big model to expand the problem, change the expression method but not the meaning, and get the expanded problem; Verify the expanded questions to ensure that the expanded semantics are consistent with the original semantics; Provide the expanded questions, table structure information and relevant business background knowledge to the big model to generate the corresponding SQL statements; Compare the SQL execution results generated by the expanded question with the SQL execution results of the seed question, retain the questions and SQL with consistent results, and obtain the initial generalized question data.

8. The system for constructing NL2SQL data sets in a large model scenario according to claim 6, characterized in that: The steps of constructing the template question dataset in the data set construction module include: Business personnel provide problem descriptions in real scenarios, including multiple categories and dimensions; Input relevant questions, table structure information and some data into the big model, design and build prompts, and call the big model to automatically generate SQL statements that conform to syntax and semantics; Execute SQL, check and correct the incorrect SQL statements, and obtain the initial template problem and SQL statements; Instantiate the initial template problem, change it into a template form, and generate new sample data; After the splicing is completed, it is input into the database to execute the query, retain the correct questions and SQL data sets, and obtain the verified standardized template question data set.

9. The system for constructing NL2SQL data sets in a large model scenario according to claim 6, characterized in that: The data set construction module generates an initial seed data set by using the problem statement, table structure information and partial data provided by the business personnel before the data set is constructed.

10. The system for constructing NL2SQL data sets in a large model scenario according to claim 9, characterized in that: The data set construction module uses a large model to expand the questions in the initial seed data set to obtain expanded questions; verifies the expanded questions to ensure that the expanded semantics are consistent with the original semantics; and uses the large model to generate SQL statements corresponding to the expanded questions to form initial generalized question data.

Citation Information

Patent Citations

  • Automatic reply method and device for user inquiry and electronic equipment

    CN113254619A

  • Method for automatically generating database query statement based on NLP language model

    CN116991869A

  • Method for generating statement from natural language to SQL in business scene based on large model

    CN117708162A

  • Method and system for automatically generating sql and replying questions

    CN118093634A

  • Medical insurance intelligent query method and system based on NL2SQL

    CN118132579A

Cited By

  • Supervision fine tuning method for Text2SQL task in low-resource scene

    CN119474277A

  • Method and device for realizing NL2SQL based on operation and maintenance industry large model

    CN119961285A