Method, device, equipment and medium for converting natural language into database statements
By fine-tuning the instructions of generative large models and optimizing the training set, the problems of slow training speed and low conversion efficiency in existing technologies are solved, achieving efficient and accurate conversion of natural language into database query statements, and improving the convenience and intelligence of database operations.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-15
- Publication Date
- 2026-03-17
AI Technical Summary
Existing large-scale language models suffer from slow training speed, low conversion efficiency, and low conversion accuracy in natural language to SQL statement conversion.
Generative large models are used for instruction fine-tuning. By constructing an instruction fine-tuning training set, combining positional encoding and RMSnorm layer standardization, Chatgpt3.5 is used to generate questions and query statements, and the model is trained and optimized through Byte-Pair Encoding and transformer architecture.
It improves training speed and conversion accuracy, achieves efficient and accurate conversion of natural language to database query statements, reduces annotation costs, and enhances the convenience and intelligence of database operations.
Smart Images
Figure CN117370378B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method, apparatus, device, and medium for converting natural language into database statements based on a generative large model. Background Technology
[0002] SQL is a programming language specifically designed for managing and manipulating databases. In information retrieval and database queries, writing queries in SQL can be quite challenging for non-technical professionals. Therefore, existing technologies utilize Text-to-SQL for language conversion and information retrieval. Text-to-SQL refers to the conversion of natural language into Structured Query Language (SQL). By automatically converting user-input natural language, such as various user questions, into SQL queries that can be manipulated by the database, it enables database-based automated question-and-answer capabilities and retrieves the required information from the database. Text-to-SQL has wide applications in areas such as intelligent customer service, improving customer service response efficiency and, to some extent, reducing the workload of human customer service representatives.
[0003] Currently, large language models (such as GPT-4) are commonly used for converting natural language to SQL statements. Compared to small and medium-sized language models, they require fewer training samples, have lower computational demands, and are less prone to overfitting. These large language models can achieve automatic SQL statement conversion after training with zero-shot, few-shot hints, and context learning. However, existing large language models still suffer from slow training speed, low conversion efficiency, and low conversion accuracy. Summary of the Invention
[0004] To overcome the shortcomings of existing technologies, one of the objectives of this invention is to provide a method for converting natural language into database statements, which uses a generative large model to fine-tune the constructed training set to obtain a high-accuracy SQL statement conversion large model.
[0005] One of the objectives of this invention is achieved through the following technical solution:
[0006] A method for converting natural language into database statements includes the following steps:
[0007] Based on the database table structure information and the prompt learning template for generating question-and-answer pairs, Chatgpt 3.5 generates the corresponding questions and query statements.
[0008] Based on the question and query statement, construct a fine-tuning training set for instructions. The training set includes a first training set with the question, database table name, and column name as inputs and the relevant table name involved in the question as outputs, and a second training set with the question, database table name, and SQL statement corresponding to the question as inputs and outputs.
[0009] The training set is fine-tuned according to the instructions, and the generative large model is fine-tuned by combining positional encoding and RMSnorm layer standardization to obtain a model that converts natural language into database statements.
[0010] Receive the text of the query question and input the model, and output the corresponding database query statement.
[0011] Furthermore, the database table structure information includes table name, column name, primary key, and foreign key.
[0012] Furthermore, a training set for instruction fine-tuning is constructed based on the aforementioned question and query statement, including:
[0013] The database table name column is embedded into the batch generation of questions and corresponding database hint learning templates. Using chatgpt, questions and corresponding database query statements are generated in batches to obtain the initial dataset.
[0014] The first training set is obtained by embedding the question and database table name as inputs and the table name corresponding to the database query language as labels into the prompt learning template.
[0015] Using questions and database table and column names as input, and the corresponding database query language as labels, the second training set is obtained by embedding them into the prompt learning template.
[0016] The first training set and the second training set are randomly combined to obtain the instruction fine-tuning training set.
[0017] Furthermore, fine-tuning of instructions is achieved through semantic parsing and generation capabilities of generative large models, including:
[0018] The instruction fine-tuning training set is segmented using Byte-Pair Encoding;
[0019] The segmented instruction fine-tuning training set is input into a generative large model based on the transformer architecture for model training.
[0020] Model optimization is achieved through positional encoding and RMSnorm layer normalization, and model training is accelerated using the Deepspeed framework to complete the model training.
[0021] Furthermore, the transformer architecture is a decider-only structure.
[0022] Furthermore, the model training includes:
[0023] Initialize model parameters;
[0024] The training data is passed to the model, and the model's predicted output is calculated through forward propagation;
[0025] The error between the predicted output and the actual result is calculated using a loss function;
[0026] The gradients of the model parameters are calculated using the backpropagation algorithm;
[0027] Update the model parameters based on the gradient calculation results;
[0028] Repeat the training iterations until the error is less than the preset value, and the model training is complete.
[0029] Furthermore, the system receives the text of the query question and inputs it into the model, then outputs the corresponding database query statement, including:
[0030] Receive the text containing the query question and perform data cleaning;
[0031] The text is combined with database information and embedded into the prompt learning template, which is then used as the input value to the model.
[0032] Obtain the model output and return it to the sender of the text.
[0033] The second objective of this invention is to provide an apparatus for converting natural language into database statements.
[0034] The second objective of this invention is achieved by the following technical solution:
[0035] An apparatus for converting natural language into database statements, comprising:
[0036] The generation module is used to generate corresponding questions and query statements through Chatgpt 3.5 based on the database table structure information and the prompt learning template for generating question-answer pairs.
[0037] The training module is used to construct a fine-tuning training set based on the question and query statement. The training set includes a first training set with the question and database table names as inputs and the table names corresponding to the database query language as labels, and a second training set with the question and database table names as inputs and the corresponding database query language as labels. The module then fine-tunes the training set based on the instructions and performs generative large-scale model instruction fine-tuning by combining positional encoding and RMSnorm layer standardization to obtain a model that converts natural language into database statements.
[0038] The processing module is used to receive the text of the query question, input it into the model, and output the corresponding database query statement.
[0039] A third objective of this invention is to provide an electronic device that performs one of the objectives of the invention, comprising a processor, a storage medium, and a computer program, wherein the computer program is stored in the storage medium and, when executed by the processor, implements the aforementioned method for converting natural language into database statements.
[0040] A fourth objective of this invention is to provide a computer-readable storage medium storing one of the objectives of the invention, wherein a computer program is stored thereon, and when the computer program is executed by a processor, it implements the above-described method for converting natural language into database statements.
[0041] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0042] This invention significantly reduces annotation costs by designing prompt learning templates for question-and-answer pairs and combining them with Chatgpt 3.5 to generate questions and SQL query statements. Furthermore, this invention utilizes the generated question-and-answer pairs to design two types of instruction fine-tuning prompt templates: one for filtering relevant database content and the other for generating database query language. A generative large-scale model is then used to fine-tune the instructions on the constructed training set. This not only accelerates training speed and improves training accuracy but also more efficiently converts natural language into SQL query statements, making database operations more convenient and intelligent. Attached Figure Description
[0043] Figure 1 This is a flowchart of the method for converting natural language into database statements in Implementation Example 1;
[0044] Figure 2 This is a block diagram of the device for converting natural language into database statements in Embodiment 2.
[0045] Figure 3 This is a structural block diagram of the electronic device in Embodiment 3. Detailed Implementation
[0046] The present invention will now be described in more detail with reference to the accompanying drawings. It should be noted that the following description of the present invention with reference to the accompanying drawings is merely illustrative and not restrictive. Various embodiments can be combined with each other to form other embodiments not shown in the following description.
[0047] Example 1
[0048] Example 1 provides a method for converting natural language into database statements. It aims to improve the accuracy of the natural language to database statement model by processing data and fine-tuning a large model. This can meet the actual needs of database queries in industry and make database queries more common and easier to use.
[0049] This embodiment uses a model with powerful semantic parsing and generative capabilities to complete the Text-to-SQL task. The task is broken down into two sub-tasks: database content selection and database query language generation. Prompts are constructed for each sub-task to form a Text-to-SQL instruction fine-tuning training set. Utilizing contextual hint learning, a large generative model is trained to possess both database content selection and database query language capabilities, providing users with automatic, efficient, and accurate SQL retrieval.
[0050] Based on the above principles, please refer to Figure 1 As shown, a method for converting natural language into database statements includes the following steps:
[0051] S1. Based on the database table structure information, generate a question-and-answer pair prompt learning template, and generate the corresponding questions and query statements using Chatgpt3.5;
[0052] The database table structure information in S1 includes table name, column name, primary key, and foreign key.
[0053] The aforementioned ChatGPT is a massive natural language processing (NLP) model developed by OpenAI. Its algorithm is based on Transformer models, pre-training techniques, fine-tuning, and Beam Search, and utilizes a large language model trained on the GPT-3.5 architecture. It can generate coherent, natural, and fluent conversational responses through NLP and can be used in various applications such as language translation, text summarization, and speech recognition. Since ChatGPT 3.5 is existing technology, this embodiment will not elaborate on it. In S1, the batch generation of questions and query statements can be achieved simply by calling the ChatGPT 3.5 API interface.
[0054] S2. Construct a fine-tuning training set based on the question and query statement. The training set includes a first training set with the question and database table name as input, and a second training set with the question and database table name as input.
[0055] The difference between the two training sets is that one training set requires outputting the tables involved in the problem, while the other requires outputting the corresponding SQL statement to improve the model's accuracy. Specifically, the training set is fine-tuned by constructing instructions based on the problem and query statement, including:
[0056] The first training set is obtained by embedding the question and database table name as inputs and the table name corresponding to the database query language as labels into the prompt learning template.
[0057] Using questions and database table and column names as input, and the corresponding database query language as labels, the second training set is obtained by embedding them into the prompt learning template.
[0058] The first training set and the second training set are randomly combined to obtain the instruction fine-tuning training set.
[0059] The corresponding learning template above is:
[0060] "{"Instruction":"Generate the table in the question based on the table name and the question","Input":{question},"Output":{corresponding table name}}"; "{"Instruction":"Generate an SQL statement based on the table name-column name and the corresponding question provided below","Input":{question},"Output":{standard SQL statement}}".
[0061] S3. Fine-tune the training set according to the instructions, and perform generative large model instruction fine-tuning by combining position encoding and RMSnorm layer standardization to obtain a model that converts natural language into database statements.
[0062] S3 specifically includes:
[0063] Each example in the training set for fine-tuning the instruction is segmented using Byte-Pair Encoding.
[0064] The word segmentation process includes: initializing the vocabulary by using all characters as initial units, treating each character as a "word"; counting character or sub-word frequencies by traversing the corpus and counting the frequency of each character or sub-word in the text; merging the most frequent characters or sub-words by merging the most frequent characters or sub-words into a new unit and updating the vocabulary; and repeating the merging steps until a predefined vocabulary size is reached or other stopping conditions are met.
[0065] The segmented instruction fine-tuning training set is input into a generative large model based on the transformer architecture for model training.
[0066] To accelerate training and reduce model complexity, the transformer architecture described above is a decider-only structure in this embodiment. Generative models can generate natural and fluent text, and pre-trained large generative models perform excellently in understanding natural language text, providing a better understanding of context.
[0067] Model optimization is achieved through positional encoding and RMSnorm layer normalization, and model training is accelerated using the Deepspeed framework to complete the model training.
[0068] The model optimization methods described above, using positional encoding and RMSnorm layer normalization, can help suppress the gradient explosion problem in deep neural networks and are relatively simple to implement. This not only reduces model complexity but also decreases the computational costs of training and inference.
[0069] Using the DeepSpeed framework for training effectively improves the training speed of the model.
[0070] The above model training includes:
[0071] Initialize model parameters; for example, perform random initialization of model weights and other parameter initialization.
[0072] Forward propagation: The training data is passed to the model, and the model's predicted output is calculated through forward propagation;
[0073] Loss calculation: The error between the predicted output and the actual result is calculated using a loss function;
[0074] Backpropagation: The gradient of the model parameters is calculated using the backpropagation algorithm; the gradient represents the degree of influence of the loss function on each parameter.
[0075] Update model parameters: Update the model parameters based on the gradient calculation results;
[0076] Repeat the training iterations until the error is less than the preset value, and the model training is complete.
[0077] S4. Receive the text of the query question and input the model, and output the corresponding database query statement.
[0078] S4 specifically includes:
[0079] Receive the text containing the query question and perform data cleaning; remove useless words and stop words.
[0080] The text is combined with database information and embedded into the prompt learning template, which is then used as the input value to the model.
[0081] Obtain the model output and return it to the sender of the text.
[0082] In summary, the method described in this embodiment can automatically analyze user input, parse the information contained therein, and construct a corresponding database query structure based on the parsing results. By converting natural language into structured query statements, it can quickly retrieve relevant information from the database. The query results will be accurately extracted and presented to the user in an easy-to-read manner, ensuring that the user can quickly obtain the data they need.
[0083] Example 2
[0084] Example 2 discloses an apparatus corresponding to the method for converting natural language into database statements in the above embodiments. This is a virtual apparatus structure as described in the above embodiments; please refer to [link / reference]. Figure 2 As shown, it includes:
[0085] The generation module 210 is used to generate corresponding questions and query statements through Chatgpt 3.5 based on the database table structure information and the prompt learning template for generating question-answer pairs.
[0086] Training module 220 is used to construct an instruction fine-tuning training set based on the question and query statement. The training set includes a first training set with the question and database table name column names as inputs and the table names corresponding to the database query language as outputs, and a second training set with the question and database table names as inputs and the SQL statement corresponding to the question as outputs. The training set is fine-tuned according to the instructions, and the generative large model is fine-tuned by combining position encoding and RMSnorm layer standardization to obtain a model that converts natural language into database statements.
[0087] The processing module 230 is used to receive the text of the query question, input the model, and output the corresponding database query statement.
[0088] Preferably, constructing a training set for instruction fine-tuning based on the question and query statement includes:
[0089] The first training set is obtained by embedding the question and database table name as inputs and the table name corresponding to the database query language as labels into the prompt learning template.
[0090] Using questions and database table and column names as input, and the corresponding database query language as labels, the second training set is obtained by embedding them into the prompt learning template.
[0091] The first training set and the second training set are randomly combined to obtain the instruction fine-tuning training set.
[0092] Preferably, fine-tuning of instructions is performed through semantic parsing and generation capabilities of generative large models, including:
[0093] The instruction fine-tuning training set is segmented using Byte-Pair Encoding;
[0094] The segmented instruction fine-tuning training set is input into a generative large model based on the transformer architecture for model training.
[0095] Model optimization is achieved through positional encoding and RMSnorm layer normalization, and model training is accelerated using the Deepspeed framework to complete the model training.
[0096] Preferably, the system receives the text of the query question and inputs it into the model, then outputs the corresponding database query statement, including:
[0097] Receive the text containing the query question and perform data cleaning;
[0098] The text is combined with database information and embedded into the prompt learning template, which is then used as the input value to the model.
[0099] Obtain the model output and return it to the sender of the text.
[0100] Example 3
[0101] Figure 3 This is a schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention, as shown below. Figure 3 As shown, the electronic device includes a processor 310, a memory 320, an input device 330, and an output device 340; the number of processors 310 in the computer device can be one or more. Figure 3 Taking a processor 310 as an example; the processor 310, memory 320, input device 330, and output device 340 in the electronic device can be connected via a bus or other means. Figure 3 Taking the example of a connection between China and Israel via a bus.
[0102] The memory 320, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the method of converting natural language into database statements in this embodiment of the invention. The processor 310 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 320, thereby implementing the method of converting natural language into database statements as described in Embodiment 1 above.
[0103] The memory 320 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 320 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 320 may further include memory remotely located relative to the processor 310, which can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0104] Input device 330 can be used to receive input user identity information, database information data, etc. Output device 340 may include display devices such as a display screen.
[0105] Example 4
[0106] Embodiment 4 of the present invention also provides a storage medium containing computer-executable instructions, which can be used by a computer to execute a method for converting natural language into database statements, the method comprising:
[0107] Based on the database table structure information, generate a question-and-answer pair prompt learning template, and generate the corresponding questions and query statements through Chatgpt 3.5;
[0108] A fine-tuning training set is constructed based on the question and query statement. The training set includes a first training set with the question and database table name as input and the table name corresponding to the database query language as output, and a second training set with the question and database table name as input and the SQL statement corresponding to the question as output.
[0109] The training set is fine-tuned according to the instructions, and the generative large model is fine-tuned by combining positional encoding and RMSnorm layer standardization to obtain a model that converts natural language into database statements.
[0110] Receive the text of the query question and input the model, and output the corresponding database query statement.
[0111] Of course, the computer-executable instructions provided in the embodiments of the present invention are not limited to the method operations described above, but can also perform related operations in the method based on converting natural language into database statements provided in any embodiment of the present invention.
[0112] Based on the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause an electronic device (which may be a mobile phone, personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.
[0113] It is worth noting that in the embodiments of the above-described method and apparatus for converting natural language into database statements, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.
[0114] For those skilled in the art, various other corresponding changes and modifications can be made based on the technical solutions and concepts described above, and all such changes and modifications should fall within the protection scope of the claims of this invention.
Claims
1. A method of converting natural language into database statements, characterized by, The method comprises the following steps: According to the database table structure information, combine the prompt learning template for generating question and answer pairs, and generate corresponding questions and query statements through Chatgpt3.5; According to the question and query statement, an instruction fine-tuning training set is constructed, which includes a first training set with question, all database table name column names as input, and related table names involved in the question as output, and a second training set with question, related database table name as input, and SQL statement corresponding to the question as output; According to the question and query statement, an instruction fine-tuning training set is constructed, which includes: Embed the database table name column into the prompt learning template for batch generating questions and corresponding database, use chatgpt to batch generate questions and corresponding database query statements, and obtain an initial data set; Embed the question and database table name column name into the prompt learning template as input and the table name corresponding to the database query language as label to obtain the first training set; Embed the question and database table name column name into the prompt learning template as input and the corresponding database query language as label to obtain the second training set; Randomly combine the first training set and the second training set to obtain the instruction fine-tuning training set; Fine-tune the instruction through the semantic analysis and generation capability of the generative large model, including: Carry out word segmentation on the instruction fine-tuning training set through Byte-Pair Encoding; Input the segmented instruction fine-tuning training set into the generative large model with a transformer architecture as the base for model training; the transformer architecture is a deocder-only structure; Optimize the model through position encoding and RMSNorm layer standardization, accelerate the model training through the deepspeed framework, and complete the model training; According to the instruction fine-tuning training set, combine position encoding and RMSNorm layer standardization to fine-tune the generative large model, and obtain a model for converting natural language into database statements; Receive the text of the question to be queried, input the model, and output the corresponding database query statement.
2. The method of converting natural language into database statements as recited in claim 1, wherein, The database table structure information includes table name, column name, primary key, and foreign key.
3. The method of converting natural language to database statements as recited in claim 1, wherein, The training process of the model includes: Initialize the model parameters; Pass the training data to the model, calculate the predicted output of the model through forward propagation; Calculate the error value of the predicted output and the actual result through the loss function; Calculate the gradient of the model parameters through the back propagation algorithm; Update the model parameters according to the calculation result of the gradient; Repeat the training iteration until the error is less than the preset value, and complete the model training.
4. The method of converting natural language to database statements as recited in claim 1, wherein, Receive the text of the question to be queried, input the model, and output the corresponding database query statement, including: Receive the text of the question to be queried, and perform data cleaning; Combine the text with the database information and embed it into the prompt learning template as the input value of the model and input the model; Get the model output result and return it to the sender of the text.
5. An apparatus for converting natural language into database statements, characterized by: It includes: The generating module is configured to generate corresponding questions and query statements through Chatgpt3.5 according to the database table structure information and in combination with a prompt learning template for generating question and answer pairs. The training module is configured to construct an instruction fine-tuning training set according to the questions and query statements, the training set including a first training set with questions and database table name and column names as inputs and related table names involved in the questions as outputs, and a second training set with questions and database table names as inputs and SQL statements corresponding to the questions as outputs; and perform generative large model instruction fine-tuning in combination with position encoding and RMSNorm layer standardization according to the instruction fine-tuning training set, to obtain a model for converting natural language into database statements. The training module is configured to construct an instruction fine-tuning training set according to the questions and query statements, the training set including a first training set with questions and database table name and column names as inputs and related table names involved in the questions as outputs, and a second training set with questions and database table names as inputs and SQL statements corresponding to the questions as outputs; and perform generative large model instruction fine-tuning in combination with position encoding and RMSNorm layer standardization according to the instruction fine-tuning training set, to obtain a model for converting natural language into database statements. The training module is configured to construct an instruction fine-tuning training set according to the questions and query statements, the training set including a first training set with questions and database table name and column names as inputs and related table names involved in the questions as outputs, and a second training set with questions and database table names as inputs and SQL statements corresponding to the questions as outputs; and perform generative large model instruction fine-tuning in combination with position encoding and RMSNorm layer standardization according to the instruction fine-tuning training set, to obtain a model for converting natural language into database statements. The training module is configured to construct an instruction fine-tuning training set according to the questions and query statements, the training set including a first training set with questions and database table name and column names as inputs and related table names involved in the questions as outputs, and a second training set with questions and database table names as inputs and SQL statements corresponding to the questions as outputs; and perform generative large model instruction fine-tuning in combination with position encoding and RMSNorm layer standardization according to the instruction fine-tuning training set, to obtain a model for converting natural language into database statements. The training module is configured to construct an instruction fine-tuning training set according to the questions and query statements, the training set including a first training set with questions and database table name and column names as inputs and related table names involved in the questions as outputs, and a second training set with questions and database table names as inputs and SQL statements corresponding to the questions as outputs; and perform generative large model instruction fine-tuning in combination with position encoding and RMSNorm layer standardization according to the instruction fine-tuning training set, to obtain a model for converting natural language into database statements. The processing module is configured to receive a text of a question to be queried and input the text into the model to output a corresponding database query statement. The computer program is configured to implement the method for converting natural language into database statements according to any one of claims 1 to 4 when executed by a processor. The computer program is configured to implement the method for converting natural language into database statements according to any one of claims 1 to 4 when executed by a processor. 6. An electronic device comprising a processor, a storage medium, and a computer program stored in the storage medium, characterized in that, 7. A computer readable storage medium having stored thereon a computer program, characterized in that,