A method for converting natural language sentences into SQL sentences

Abstract

The application belongs to the technical field of natural language processing, and relates to a natural language sentence to SQL sentence conversion method, which comprises the following steps: 1) fine-tuning a large language model; 2) obtaining a natural language sentence to be converted; 3) inputting the natural language sentence into the fine-tuned large language model to obtain a converted SQL sentence. The application solves the problem that the fine-tuning method based on a large language model is prone to overfitting when applied to natural language sentence to SQL sentence conversion, thereby improving the overall effect of natural language sentence to SQL sentence conversion.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology, and relates to a method for converting natural language statements into SQL statements, and more particularly to a method for converting natural language statements into SQL statements that combines a circular consistency mechanism and fine-tuning of a large language model. Background Technology

[0002] Deep learning methods were previously commonly used to translate natural language statements into SQL statements. However, traditional deep learning methods lack general natural language understanding capabilities, require large amounts of labeled data, have weak cross-scenario capabilities, low flexibility, and offer virtually no room for adjustment once the model is deployed without retraining.

[0003] With the rapid development of artificial intelligence and big data technologies, the new generation of artificial intelligence technologies, represented by large language models, has brought revolutionary breakthroughs. How to reconstruct and upgrade traditional products and businesses based on large language models is a hot research topic.

[0004] Natural language (NLP) translation into SQL is one of the most popular research areas in natural language processing. Human natural language expression is highly diverse, and traditional methods struggle to handle it effectively. Large language models, however, possess powerful emergent capabilities, demonstrating a general understanding of natural language with high cross-domain capabilities and flexibility. Therefore, NLP-based methods for translating NLP into SQL have become a hot research topic.

[0005] For example, the paper "Can LLM Already Serve as A Database Interface? A Big Bench for Large-Scale Database Grounded Text-to-SQLs" published by Alibaba DAMO Academy (arXiv.2023.05.03111) proposes a method for converting natural language to SQL statements based on large-scale model fine-tuning and a new natural language to SQL dataset. The specific steps are as follows: S1, obtain the natural language question input by the user; S2, input the natural language question into a pre-tuned large language model to obtain the corresponding SQL statement. The large language model uses T5, and the fine-tuning training method uses full-parameter fine-tuning. The training data is constructed by concatenating the user-input natural language question and database table information to form a complete input, and the corresponding SQL statement is the output.

[0006] The aforementioned natural language to SQL method based on fine-tuning of a large language model essentially generates a distribution that approximates SQL statements. However, it lacks a sufficient understanding of the meaning and syntax of SQL statements, making it prone to overfitting during fine-tuning training and limiting its accuracy.

[0007] Therefore, in order to address the shortcomings of the existing technologies, it is necessary to develop a new method for converting natural language statements to SQL statements. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention proposes a method for converting natural language statements to SQL statements. By introducing a cycle consistency alignment mechanism, it solves the problem of overfitting when applying the method based on large language model fine-tuning to convert natural language statements to SQL statements, thereby improving the overall effect of converting natural language statements to SQL statements.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] A method for converting natural language statements to SQL statements, characterized by the following steps:

[0011] 1) Fine-tune the training of the large language model;

[0012] 2) Obtain the natural language statement to be converted;

[0013] 3) Input the natural language statement into the finely tuned large language model to obtain the converted SQL statement.

[0014] Preferably, step 1) specifically includes:

[0015] 1.1) Create a training dataset for converting natural language statements to SQL statements;

[0016] 1.2) Create a training dataset for converting SQL statements to natural language statements;

[0017] 1.3) Use the SQL statements converted to natural language statements to train the dataset and then fine-tune the large language model in reverse;

[0018] 1.4) Use the natural language statement to SQL statement training dataset to perform cycle consistency alignment fine-tuning on the reverse-fine-tuned large language model.

[0019] Preferably, in step 1.4), the loss function used during the cyclic consistency alignment fine-tuning is denoted as L. final ,but

[0020] L final =τ*L original +(1-τ)*L cycle

[0021]

[0022] Among them, L original L represents the original loss function of the large language model;cycle The cycle consistency loss function term is represented using cross-entropy; step_total represents the total number of training iterations; step_current represents the current iteration step; τ represents the adjustable parameter of the loss function term; and δ is the cutoff coefficient.

[0023] Preferably, Where n represents the total number of data samples in the natural language to SQL statement training dataset, x i Let p(x) represent the i-th data sample. i ) represents the true probability distribution of the i-th data sample, q(x) i ) represents the predicted probability distribution of the i-th data sample.

[0024] Preferably, δ = 0.5.

[0025] Preferably, in steps 1.3) and 1.4), a full-parameter fine-tuning method or a LoRA fine-tuning method is used.

[0026] Preferably, in step 1.1), the natural language statement to SQL statement training dataset is derived from an open-source dataset, and different open-source datasets are processed into a unified format to obtain the natural language statement to SQL statement training dataset.

[0027] Preferably, in step 1.2), the SQL statement to natural language statement training dataset is constructed by reverse engineering the natural language statement to SQL statement training dataset, that is, by swapping the positions of the SQL statements and natural language statements.

[0028] Preferably, the finely tuned large language model is deployed on a cloud server.

[0029] Preferably, the large language model is ChatGLM, ChatGPT, or GPT-4.

[0030] Compared with the prior art, the natural language statement to SQL statement method of the present invention has one or more of the following beneficial technical effects:

[0031] 1. This invention solves the problem of overfitting when applying the method of fine-tuning based on large language models to convert natural language statements to SQL statements by introducing a cycle consistency alignment mechanism, thereby improving the overall effect of converting natural language statements to SQL statements.

[0032] 2. In this invention, while generating SQL statements using a large language model, the generated SQL statements are reversed to generate natural language statements, forming a circular consistency alignment mechanism such as converting natural language statements to SQL statements and SQL statements to natural language statements. This mechanism enables the large language model not only to accurately convert natural language statements to SQL statements, but also to parse SQL statements into natural language statements, allowing the large language model to fully understand the meaning and syntax of SQL statements.

[0033] 3. This invention is applied to intelligent data interaction systems, which can lower the threshold for using database interaction. Attached Figure Description

[0034] Figure 1 This is a flowchart of the natural language statement to SQL statement method of the present invention.

[0035] Figure 2 This is a schematic diagram of the cyclic consistency alignment mechanism used in this invention.

[0036] Figure 3 This is a flowchart of the fine-tuning training of the large language model in this invention. Detailed Implementation

[0037] The present invention will be further described below with reference to the accompanying drawings and embodiments. The content of the embodiments is not intended to limit the scope of protection of the present invention.

[0038] To address the existing problems in converting natural language statements to SQL statements, this patent provides a method that combines a cycle consistency alignment mechanism with large language model fine-tuning. This solves the overfitting problem that easily occurs when applying large language model fine-tuning methods to convert natural language statements to SQL statements, thereby improving the overall performance of converting natural language statements to SQL statements.

[0039] Figure 1 A flowchart of the natural language statement to SQL statement method of the present invention is shown. Figure 1 As shown, the natural language statement to SQL statement conversion method of the present invention includes the following steps:

[0040] I. Fine-tuning the training of the large language model.

[0041] The large language model mentioned is an existing model in the industry, such as ChatGLM, ChatGPT, or GPT-4. By fine-tuning the large language model during training, it is made able to better understand natural language statements and generate corresponding SQL statements.

[0042] In this invention, a cycle consistency alignment mechanism is introduced during fine-tuning training of a large language model. That is, as follows: Figure 2As shown, while the large language model generates SQL statements, it reverses the generated SQL statements to generate natural language statements, forming a circular consistency alignment mechanism such as converting natural language statements to SQL statements and SQL statements to natural language statements. This mechanism enables the large language model not only to accurately convert natural language statements into SQL statements, but also to parse SQL statements into natural language statements, allowing the large language model to fully understand the meaning and syntax of SQL statements.

[0043] Specifically, such as Figure 2 As shown, the fine-tuning training of the large language model of the present invention mainly includes the following steps:

[0044] 1. Create a training dataset for converting natural language statements to SQL statements.

[0045] To fine-tune the training of a large language model, you must first obtain a training dataset of natural language statements converted to SQL.

[0046] In this invention, the natural language statement to SQL statement training dataset is derived from open-source datasets, primarily including WikiSQL and Spider. Furthermore, to facilitate fine-tuning of large language models, different open-source datasets need to be processed into a unified format.

[0047] In this invention, the format of the natural language statement to SQL statement training dataset is as follows:

[0048] {"input":"You are a natural language to SQL generator. It directly outputs SQL statements based on the natural language, without interpretation. Table names and fields are as follows:\n#table('Representative','Years','State','Party','Lifespan')\nInput natural language:"Which state does Jimmy Quillen represent?"\nOutput SQL:","target":"SELECT State FROM table WHERE Representative=jimmyquillen"}

[0049] Here, "input" represents the complete input, including prompt and the natural language statement to be converted, and "target" represents the output SQL statement.

[0050] 2. Create a training dataset for converting SQL statements to natural language statements.

[0051] In this invention, the SQL statement to natural language statement training dataset is constructed by reverse engineering the natural language statement to SQL statement training dataset, that is, by swapping the positions of the SQL statements and natural language statements.

[0052] In this invention, the format of the SQL statement to natural language statement training dataset is as follows:

[0053] {"input":"You are a SQL-to-natural language generator that directly outputs natural language based on SQL statements, without interpretation. Table names and fields are as follows:\n#table('Representative','Years','State','Party','Lifespan')\nInput SQL:"SELECT State FROM table WHERE Representative=jimmy quillen"\nOutput natural language:","target":"Which state does Jimmy Quillenrepresent?"}

[0054] Here, "input" represents the complete input, including prompt and the SQL statement to be converted, and "target" represents the output natural language statement.

[0055] 3. Use SQL statements to convert natural language statements to train datasets for reverse fine-tuning of large language models.

[0056] In other words, the original large language model is used to perform reverse fine-tuning of the SQL statement to natural language statement conversion on the SQL statement to natural language statement training dataset created above, so as to enhance the large language model's ability to understand and parse SQL statements.

[0057] Among them, the fine-tuning method can use mainstream large language model fine-tuning methods, such as full parameter fine-tuning method, LoRA method, etc.

[0058] 4. Use the natural language statement to SQL statement training dataset to perform cycle consistency alignment fine-tuning on the reverse-fine-tuned large language model.

[0059] In other words, based on the large language model that has undergone reverse fine-tuning after converting SQL statements to natural language statements, cycle consistency alignment fine-tuning is performed. Specifically, the large language model after reverse fine-tuning is used on the natural language statement to SQL statement training dataset created above to perform cycle consistency alignment fine-tuning.

[0060] Among them, the fine-tuning method can use mainstream large language model fine-tuning methods, such as full parameter fine-tuning method, LoRA method, etc.

[0061] In this invention, the loss function used for cyclic consistency alignment fine-tuning is denoted as L. final Because of the introduction of a circular consistency alignment mechanism,

[0062] L final =τ*L original +(1-τ)*L cycle

[0063]

[0064] Among them, L original L represents the original loss function of the large language model; cycle The cycle consistency loss function term is represented by cross-entropy; step_total represents the total number of training iterations; step_current represents the current iteration step; τ represents an adjustable parameter of the loss function term. The larger the value, the more dominant the corresponding natural language statement to SQL statement task is on the overall loss function; δ is the cutoff coefficient, used to control the lower limit of the original loss function calculation. The parameter setting can be referenced as δ = 0.5.

[0065] Cross-entropy is the cross-entropy between the natural language statement generated by the large language model and the actual natural language statement. It is used to measure the difference between two probability distributions (usually referring to the difference between the prediction result of the large language model and the actual value). The value range is [0, +∞], and the formula is defined as follows: Where x i Let p(x) represent the i-th data sample. i ) represents the true probability distribution, q(x) i ) represents the predicted probability distribution.

[0066] Therefore, in this invention, Where n represents the total number of data samples in the natural language to SQL statement training dataset, x i Let p(x) represent the i-th data sample. i ) represents the true probability distribution of the i-th data sample, q(x) i ) represents the predicted probability distribution of the i-th data sample.

[0067] Through the aforementioned reverse fine-tuning and cycle consistency alignment fine-tuning, the fine-tuning training of the large language model was completed. The fine-tuned large language model can then be deployed on a cloud server for easy use.

[0068] 2. Obtain the natural language statement to be converted.

[0069] The natural language statement to be converted is obtained and denoted as Q.

[0070] 3. Input the natural language statement into the finely tuned and trained large language model to obtain the converted SQL statement.

[0071] The natural language statement to be converted is input into the fine-tuned large language model in a question-and-answer manner. The fine-tuned large language model will automatically generate the converted SQL statement, which can be denoted as A.

[0072] The natural language (NLP) to SQL conversion method of this invention solves the overfitting problem that easily occurs when applying large language model fine-tuning methods to NLP to SQL conversion by introducing a cycle consistency alignment mechanism during the fine-tuning training of a large language model and optimizing the loss function. This improves the overall performance of NLP to SQL conversion. Furthermore, it can be applied to intelligent data interaction systems, thereby lowering the barrier to entry for database interaction.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.

Claims

1. A method for converting natural language statements to SQL statements, characterized in that, The method comprises the following steps: 1) fine-tuning a large language model; 2) obtaining a natural language sentence to be converted; 3) inputting the natural language sentence into the fine-tuned large language model to obtain a converted SQL sentence; wherein the step 1) specifically comprises: 1.1) creating a natural language sentence to SQL sentence training dataset; 1.2) creating a SQL sentence to natural language sentence training dataset; 1.3) using the SQL sentence to natural language sentence training dataset to perform reverse fine-tuning on the large language model; 1.4) using the natural language sentence to SQL sentence training dataset to perform cyclic consistency alignment fine-tuning on the reverse fine-tuned large language model; And, in the step 1.4), when performing the cycle consistency alignment fine-tuning, the loss function used is denoted as Then wherein, represents the original loss function of the large language model; represents a cycle consistency loss function term, which is expressed using cross-entropy; represents the total number of training iterations; represents the current iteration step number; represents an adjustable parameter of the loss function term; is a cut-off coefficient; wherein n represents the total number of data samples in the natural language sentence to SQL sentence training data set, represents the i th data sample, represents the true probability distribution of the i th data sample, represents the predicted probability distribution of the i th data sample.

2. The natural language sentence to SQL sentence method of claim 1, wherein, 。 3. The natural language sentence to SQL sentence method of claim 2, wherein, in the steps 1.3) and 1.4), a full parameter fine-tuning method or a LoRA fine-tuning method is adopted.

4. The natural language sentence to SQL sentence method of claim 3, wherein, in the step 1.1), the natural language sentence to SQL sentence training dataset is derived from an open source dataset, and different open source datasets are processed into a unified format to obtain the natural language sentence to SQL sentence training dataset.

5. The natural language sentence to SQL sentence method of claim 4, wherein, in the step 1.2), the SQL sentence to natural language sentence training dataset is reversely constructed from the natural language sentence to SQL sentence training dataset, i.e., the SQL sentence and the natural language sentence positions are interchanged.

6. The natural language sentence to SQL sentence method of any one of claims 1-5, wherein, The fine-tuned large language model is deployed on a cloud server.

7. The natural language sentence to SQL sentence method of claim 6, wherein, The large language model is ChatGLM, ChatGPT or GPT-4.

Images