Natural language to SQL query method and system based on execution result inspection

Abstract

The invention discloses a natural language to SQL query method and system based on execution result inspection, and the method comprises the steps: inputting a natural language problem into a Text-to-SQL data set, and selecting a plurality of examples similar to the natural language problem and corresponding SQL queries from the Text-to-SQL data set; the SQL questions and the natural language questions are input into an LLM together, and the LLM outputs candidate SQL queries and generation reasons of the candidate SQL queries; executing the candidate SQL query, inputting the obtained execution result, the natural language problem and the candidate SQL query into the trained test model to judge the correctness of the execution result, and giving out the correctness probability of the execution result; and according to the probability of generating the candidate SQL query by the LLM and the correct probability of the execution result, calculating the correct probability of the candidate SQL query, sorting the correct probability of the candidate SQL query in a descending order, and outputting the first candidate SQL query. According to the method, grammar errors of SQL query can be found, correctness of SQL query can be judged more visually, and finding of complex problems and errors existing in complex SQL query is facilitated.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a natural language to SQL query conversion method and system based on execution result verification. Background Art

[0002] Text-to-SQL makes it possible to query data using natural language by generating SQL queries from natural language problem descriptions. However, the accuracy of SQL queries generated by existing methods is still not high. Most methods directly generate SQL queries in one go based on natural language questions, but it is difficult to generate accurate SQL queries in one go for complex questions. In order to improve the accuracy of SQL queries generated using natural language, it is necessary to verify the generated SQL queries. Currently, the verification of SQL queries generated from natural language mainly determines whether the SQL queries are correct semantically based on questions and database schemas. This method makes it difficult to find errors in complex SQL queries.

[0003] Chinese patent CN116521711A - A Text-to-SQL method, system and medium. It aims at the problem of long length and the difficulty of model memory information and weak information correspondence ability in the database structure. It solves the problem of difficulty in memory information and weak information correspondence ability through information marking and application of SQL grammar rules, thereby solving the problem of long length and complex database structure in the prior art, and the problem of low accuracy when generating SQL, thereby improving the accuracy of SQL generation.

[0004] Chinese Patent CN117931841A - A language conversion method and device based on data enhancement and large language model. Data enhancement technology is used to improve the quantity, diversity and quality of training data; in order to improve the effect of model prediction, the Text-to-SQL task is split into steps, and each step is solved one by one using a large model to obtain the SQL query statement corresponding to the natural language. At the same time, targeted optimization is also carried out for different stages.

[0005] Chinese Patent CN117931851A - A Text-to-SQL method based on graph attention diffusion. A heterogeneous graph is constructed for the question and database model, and then a relational graph attention network is used to generate SQL queries through the heterogeneous graph. When constructing the heterogeneous graph, SDP semantic analysis is used to enrich the graph construction results and obtain more favorable edge relationships. In the relational graph attention network, an attention diffusion mechanism is added to diffuse the attention between single-hop nodes into the attention between multi-hop nodes, which strengthens the scope of consideration of the model and enables the model to achieve good results in cross-domain situations.

[0006] The disadvantage of the above method is that after the SQL query is generated according to the natural language question, the correctness of the generated SQL query is not verified, resulting in a low accuracy rate of the generated SQL query.

[0007] Chinese Patent CN117472958A - Method, device, equipment and medium for verifying the execution results of SQL statements. The problem that the execution status of SQL statements cannot be determined quickly and accurately by only setting verification statements. By generating verification statements based on the insert, create, update, delete, drop, and truncate statements used when building a database, and then querying the data records in the database based on the verification statements, the execution results of the SQL statements are further verified, and the grammatical problems that the database cannot recognize that appear in the statements are checked. This method requires the insert, create, update, delete, drop, truncate and other statements used to build the database, and is not suitable for verifying the execution results of the generated SQL queries in the scenario of Text-to-SQL, which generates SQL queries based on natural language questions.

[0008] Chinese Patent CN118093636A - A method for determining the correctness of SQL query statements. The main feature is to extract information from the serialized information of the input database, or to obtain candidate table column information to obtain more concise database structure information; and then determine the correctness of the SQL statement based on the user's question, candidate table column, and candidate SQL statement. This method verifies the correctness of the SQL query through the user's query question and the filtered database structure information. This verification is only a semantic level test, and it is difficult to find errors in complex questions and SQL queries. Summary of the invention

[0009] In view of the above problems, the present invention provides a natural language to SQL query method, system and storage medium based on execution result verification, aiming to verify the generated SQL query to improve the accuracy of the generated SQL query.

[0010] According to a first aspect of an embodiment of the present disclosure, a natural language to SQL query method based on execution result verification is provided, the method comprising the following steps:

[0011] Inputting the natural language question into a Text-to-SQL dataset, and selecting several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset;

[0012] Input the examples and their corresponding SQL queries and the natural language question into the LLM, and the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries;

[0013] Execute the candidate SQL query, input the obtained execution result, the natural language question, and the candidate SQL query into the trained verification model to determine the correctness of the execution result and give a correct probability of the execution result;

[0014] The correct probability of the candidate SQL query is calculated according to the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, the correct probabilities of the candidate SQL queries are arranged in descending order, and the first candidate SQL query is output.

[0015] A further technical solution of the present invention is: inputting the natural language question into a Text-to-SQL dataset, selecting several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset, and the specific implementation process includes: using word embedding technology to convert the natural language question and each natural language question in the Text-to-SQL dataset into numerical vectors respectively; calculating the distance between the natural language question and each natural language question in the Text-to-SQL dataset by Euclidean distance based on the numerical vector, and selecting the first N natural language questions in the Text-to-SQL dataset as examples after arranging them in ascending order.

[0016] A further technical solution of the present invention is: the training process of the verification model includes:

[0017] Build a dataset for training and testing the model: execute the SQL queries in the Text-to-SQL dataset in the database to obtain the execution results, and mark whether the execution results are correct;

[0018] The verification model is trained using the constructed data set: natural language questions, SQL queries and their execution results are used as inputs to the verification model, and the verification model outputs the correct probability of the execution result.

[0019] A further technical solution of the present invention is: calculating the correct probability of candidate SQL queries based on the probability of LLM generating candidate SQL queries and the correct probability of the execution results, specifically including: multiplying the generation probability of each candidate SQL query in LLM with the correct probability of the execution results obtained by the candidate SQL query in the verification model to obtain the correct probability of the candidate SQL query.

[0020] According to a second aspect of an embodiment of the present disclosure, a natural language to SQL query system based on execution result verification is provided, the system comprising:

[0021] a candidate SQL query generation module, configured to input the natural language question into a Text-to-SQL dataset, select several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset; and input the several examples and their corresponding SQL queries and the natural language question into an LLM, wherein the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries;

[0022] An SQL query verification module is used to execute the candidate SQL query, input the obtained execution result, the natural language question, and the candidate SQL query into a trained verification model to determine the correctness of the execution result and give a correct probability of the execution result;

[0023] The candidate SQL query rearrangement module is used to calculate the correct probability of the candidate SQL query according to the probability of generating the candidate SQL query by the LLM and the correct probability of the execution result, arrange the correct probabilities of the candidate SQL queries in descending order and output the first candidate SQL query.

[0024] A further technical solution of the present invention is: in the candidate SQL query generation module, the natural language question is input into a Text-to-SQL data set, and several examples similar to the natural language question and their corresponding SQL queries are selected from the Text-to-SQL data set. The specific implementation process includes: using word embedding technology to convert the natural language question and each natural language question in the Text-to-SQL data set into numerical vectors respectively; based on the numerical vector, the distance between the natural language question and each natural language question in the Text-to-SQL data set is calculated by Euclidean distance, and the first N natural language questions in the Text-to-SQL data set are selected as examples after arranging in ascending order.

[0025] A further technical solution of the present invention is: in the SQL query verification module, the training process of the verification model includes:

[0026] Build a dataset for training and testing the model: execute the SQL queries in the Text-to-SQL dataset in the database to obtain the execution results, and mark whether the execution results are correct;

[0027] The verification model is trained using the constructed data set: natural language questions, SQL queries and their execution results are used as inputs to the verification model, and the verification model outputs the correct probability of the execution result.

[0028] A further technical solution of the present invention is: in the candidate SQL query rearrangement module, the correct probability of the candidate SQL query is calculated based on the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, specifically including: multiplying the generation probability of each candidate SQL query in the LLM with the correct probability of the execution result obtained by the candidate SQL query in the verification model to obtain the correct probability of the candidate SQL query.

[0029] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the above-mentioned natural language to SQL query method based on execution result verification are implemented.

[0030] According to a fourth aspect of an embodiment of the present disclosure, a non-temporary computer-readable storage medium is provided, on which computer instructions are stored. When the instructions are executed by a processor, the steps of the above-mentioned natural language to SQL query method based on execution result verification are implemented.

[0031] The embodiments of the present disclosure provide a natural language to SQL query method, system and storage medium based on execution result verification, which generates candidate SQL queries using examples. Similar question and SQL query pairs are obtained as examples in the Text-to-SQL dataset through natural language questions. The questions, database schema and examples are provided to the large language model, which generates multiple SQL queries that can answer the provided questions based on the examples. This method can ensure the diversity of generated SQL queries and increase the probability of selecting the correct SQL query through verification.

[0032] Verify the correctness of the generated SQL query through the execution results. Use the Text-to-SQL dataset to train a verification model. The verification model judges the correctness of the execution result based on the natural language question, the generated SQL query and its execution result in the database, and outputs the probability of the execution result being correct. Using the execution results, it is easy to find that the generated SQL query has an incorrect execution result due to syntax errors. For complex questions and SQL queries, the execution results can more intuitively determine the correctness of the SQL query.

[0033] Reorder the candidate SQL queries according to the correct probability of the SQL query. Calculate the correct probability of the SQL query by combining the generation probability of the SQL query generated by the large language model and the correct probability of the execution result verification. Rearrange the candidate SQL queries in order from large to small correct probability, and output the SQL query with the highest correct probability. Adjusting the order of the candidate SQL queries according to the correctness of the execution results helps to improve the correctness of the SQL query finally output. In summary, the beneficial effects of the method of the present invention include:

[0034] At present, most Text-to-SQL methods directly output the generated SQL query and verify the correctness of the SQL query. The Text-to-SQL method for verifying the generated SQL query mainly judges the correctness of the SQL query at the semantic level through natural language questions and database patterns, and it is difficult to find errors in complex questions and complex SQL queries. The present invention:

[0035] A method for generating SQL queries by using natural language questions or examples is proposed. By generating multiple candidate SQL queries, the diversity of generated SQL queries is ensured and the possibility of correct SQL queries being present in the generated SQL queries is increased.

[0036] This paper proposes a method to verify SQL queries through the execution results of SQL queries. The execution results can more easily find the syntax errors of SQL queries, more intuitively judge the correctness of SQL queries, and help to find complex problems and errors in complex SQL queries.

[0037] A candidate SQL query re-ranking method is proposed. The candidate SQL queries are re-ranked according to the generation probability of the SQL query and the correct probability of its execution result, thereby increasing the possibility of selecting the correct SQL query and thus improving the accuracy of the Text-to-SQL method.

[0038] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the present invention.

[0040] Figure 1 It is a flowchart of a natural language to SQL query method based on execution result verification in an embodiment of the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of a natural language to SQL query system based on execution result verification in an embodiment of the present invention;

[0042] Figure 3 It is a schematic diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0044] It should be mentioned before discussing the exemplary embodiments in more detail that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the steps as sequential processes, many of the steps therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but can also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0045] Some terminology descriptions based on the embodiments of the present invention:

[0046] Text: Database query questions expressed in natural language

[0047] SQL: Structured Query Language.

[0048] Text-to-SQL: Convert natural language questions into SQL database queries.

[0049] LLM: Large Language Model.

[0050] The embodiments of the present invention provide the following embodiments for a natural language to SQL query method, system and storage medium based on execution result verification:

[0051] Example 1 is used to illustrate the natural language to SQL query method based on execution result verification, see Figure 1 , which is a flow chart of a natural language to SQL query method based on execution result verification, and the specific steps include:

[0052] S1. Input the natural language question into a Text-to-SQL dataset, and select several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset;

[0053] The specific implementation process includes: using word embedding technology to convert the natural language question and each natural language question in the Text-to-SQL dataset into numerical vectors respectively; calculating the distance between the natural language question and each natural language question in the Text-to-SQL dataset through Euclidean distance based on the numerical vector, and selecting the first N natural language questions in the Text-to-SQL dataset as examples after arranging them in ascending order.

[0054] S2, inputting the several examples and their corresponding SQL queries and the natural language question into the LLM, and the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries;

[0055] S3, executing the candidate SQL query, inputting the obtained execution result, the natural language question, and the candidate SQL query into the trained verification model to determine the correctness of the execution result and give a correct probability of the execution result;

[0056] The training process of the verification model includes:

[0057] Build a dataset for training and testing the model: execute the SQL queries in the Text-to-SQL dataset in the database to obtain the execution results, and mark whether the execution results are correct;

[0058] The verification model is trained using the constructed data set: natural language questions, SQL queries and their execution results are used as inputs to the verification model, and the verification model outputs the correct probability of the execution result.

[0059] S4. Calculate the correct probability of the candidate SQL query according to the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, arrange the correct probabilities of the candidate SQL queries in descending order and output the first candidate SQL query.

[0060] Specifically, it includes: multiplying the generation probability of each candidate SQL query in the LLM by the correct probability of the execution result obtained by the candidate SQL query in the verification model to obtain the correct probability of the candidate SQL query.

[0061] The embodiment verifies the generated SQL query through the execution result of the SQL query and selects the best SQL query as the result of Text-to-SQL according to the verification result.

[0062] Based on the Text-to-SQL architecture of execution result verification, it is not easy to generate SQL queries based on natural language. For complex problems, it is difficult to generate correct SQL queries in one go. Therefore, it is necessary to verify the generated SQL queries to improve the accuracy of generated SQL queries. Compared with judging the correctness of SQL queries semantically through natural language questions and database models, the execution results of SQL queries can more intuitively judge whether it is the answer to the question and thus determine whether the SQL query is correct.

[0063] In a specific embodiment, the process of generating SQL queries based on the Text-to-SQL of the execution result verification is as follows:

[0064] Step 1: The candidate SQL query generation module selects similar examples based on the natural language question, and provides the natural language question, database schema and examples to the large language model (LLM) to generate multiple candidate SQL queries corresponding to the question.

[0065] Step 2: The SQL query verification module executes all generated candidate SQL queries on the database corresponding to the question to obtain the execution results. The trained verification model is used to judge the correctness of the execution results based on the natural language question, the candidate SQL query and its execution results, and gives the correct probability.

[0066] Step 3: The candidate SQL query rearrangement module calculates the correct probability of the candidate SQL query according to the probability of the candidate SQL query generated by the LLM and the correct probability of the execution result of the candidate SQL query, sorts the candidate SQL queries from high to low according to the calculated correct probability, and outputs the candidate SQL query with the highest correct probability. The probability of the candidate SQL query generated by the LLM is specifically set by setting parameters so that the LLM returns the probability corresponding to the result while returning the result.

[0067] Among them, candidate SQL queries are generated based on examples. LLM's powerful ability in processing natural language enables it to perform well in Text-to-SQL. However, if LLM is only allowed to generate SQL queries based on natural language questions and database schemas (obtaining the database schema by querying the database schema in the database), LLM needs to correctly understand the questions and database models, which is often difficult for more complex problems. LLM has a strong ability to summarize and generalize. If some relevant examples can be provided to LLM, it will help LLM generate correct SQL queries. Moreover, LLM can generate multiple SQL queries at the same time, so that the generated SQL queries have better diversity.

[0068] In a specific embodiment, the process of generating a candidate SQL query based on an example is as follows:

[0069] Step 11: Use word embedding technology to convert the target natural language question and all natural language questions in the Text-to-SQL dataset into numerical vectors.

[0070] Step 12: Use the converted numerical vector to calculate the distance between the target natural language question and all natural language questions in the data set through Euclidean distance, and select N data with the smallest distance as examples.

[0071] Step 13: Provide the target natural language question, database schema and the selected N examples to LLM to generate K candidate SQL queries.

[0072] SQL query verification based on execution results. Judging the correctness of SQL queries based on natural language questions and database schemas can only be done by semantically comparing whether the generated SQL queries conform to the descriptions of natural language questions and database schemas. For complex questions and database schemas, it is difficult to accurately judge the correctness of SQL queries. However, from the execution results of SQL queries, it is very intuitive to see whether it is the expected answer to the question. If the correctness of the generated SQL queries can be judged by the correctness of the execution results of SQL queries, the effect of verifying the correctness of SQL queries can be effectively improved.

[0073] In another embodiment, the process of SQL query verification based on the execution result is as follows:

[0074] Step 21: Build a dataset for training and testing the model. Execute the SQL query in the Text-to-SQL dataset in the database to obtain the execution result, and mark whether the execution result is correct, 1 for correct and 0 for error. This is for the database to be queried for each question in the Text-to-SQL dataset.

[0075] Step 22: Use the constructed data set to train and test the model. Take the natural language question, SQL query and its execution result as input, and let it output the correct probability of the execution result.

[0076] Step 23: Verify the candidate SQL queries. For each candidate SQL query, execute it in the database to be queried by the target question to obtain the execution result, input it and the natural language question and its execution result into the verification model, and the verification model outputs the probability that its execution result is correct.

[0077] Rearrange candidate SQL queries. Each SQL query generated by LLM has its corresponding generation probability, which indicates the probability that LLM considers the SQL query to be the best answer. Usually, the SQL query with the highest generation probability is directly used as the final result. In order to improve the accuracy of the output SQL query, it is necessary to reorder the candidate SQL queries in combination with the test results of the SQL query, so that the final output SQL query not only considers its generation probability in LLM but also considers the results of the correctness test, so that the SQL query with a higher probability of correctness can be obtained.

[0078] In another embodiment, the process of re-arranging candidate SQL queries is as follows:

[0079] Step 31: Calculate the correct probability of the candidate SQL query. Multiply the probability of each candidate SQL query being generated in the LLM by the probability of the correct execution result obtained in the verification model to obtain the correct probability of the candidate SQL query.

[0080] Step 32: Re-rank the candidate SQL queries according to their correct probabilities. Arrange the K candidate SQL queries from high to low according to their correct probabilities, and output the SQL query with the highest correct probability as the final answer.

[0081] Another embodiment is used to illustrate a natural language to SQL query system based on execution result verification, see Figure 2 , the system comprising:

[0082] A candidate SQL query generation module 210 is configured to input the natural language question into a Text-to-SQL dataset, select several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset; and input the several examples and their corresponding SQL queries and the natural language question into an LLM, and the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries;

[0083] The SQL query verification module 220 is used to execute the candidate SQL query, input the obtained execution result, the natural language question, and the candidate SQL query into the trained verification model to determine the correctness of the execution result and give the correct probability of the execution result;

[0084] The candidate SQL query rearrangement module 230 is used to calculate the correct probability of the candidate SQL query according to the probability of generating the candidate SQL query by LLM and the correct probability of the execution result, arrange the correct probabilities of the candidate SQL queries in descending order and output the first candidate SQL query.

[0085] In the candidate SQL query generation module 210, the natural language question is input into a Text-to-SQL data set, and several examples similar to the natural language question and their corresponding SQL queries are selected from the Text-to-SQL data set. The specific implementation process includes: using word embedding technology to convert the natural language question and each natural language question in the Text-to-SQL data set into numerical vectors respectively; based on the numerical vector, the distance between the natural language question and each natural language question in the Text-to-SQL data set is calculated by Euclidean distance, and the first N natural language questions in the Text-to-SQL data set are selected as examples after arranging in ascending order.

[0086] In the SQL query verification module 220, the training process of the verification model includes:

[0087] Build a dataset for training and testing the model: execute the SQL queries in the Text-to-SQL dataset in the database to obtain the execution results, and mark whether the execution results are correct;

[0088] The verification model is trained using the constructed data set: natural language questions, SQL queries and their execution results are used as inputs to the verification model, and the verification model outputs the correct probability of the execution result.

[0089] In the candidate SQL query rearrangement module 230, the correct probability of the candidate SQL query is calculated based on the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, specifically including: multiplying the generation probability of each candidate SQL query in the LLM with the correct probability of the execution result obtained by the candidate SQL query in the verification model to obtain the correct probability of the candidate SQL query.

[0090] In addition to the upper module, the system 200 may also include other components. However, since these components are irrelevant to the content of the embodiment of the present disclosure, their illustration and description are omitted here.

[0091] For other specific working processes of the natural language-to-SQL query system 200 based on execution result verification, please refer to the description of the above-mentioned natural language-to-SQL query method embodiment based on execution result verification, which will not be repeated here.

[0092] Another embodiment is used to illustrate that the system of the present invention can also be used with the help of Figure 3 The architecture of the computing device shown is implemented. Figure 3 The architecture of the computing device is shown. Figure 3As shown, a computer system 310, a system bus 330, one or more CPUs 340, an input / output 320, a memory 350, etc. The memory 350 can store various data or files used for computer processing and / or communication and program instructions executed by the CPU including the natural language to SQL query method based on execution result verification of the embodiment. Figure 3 The architecture shown is only exemplary and can be adjusted according to actual needs when implementing different devices. Figure 3 One or more components in. The memory 350, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the natural language-to-SQL query method based on execution result verification in the embodiment of the present invention (for example, the candidate SQL query generation module 210, SQL query verification module 220 and candidate SQL query rearrangement module 230 in the natural language-to-SQL query system 200 based on execution result verification). One or more CPUs 340 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions and modules stored in the memory 350, that is, to implement the above-mentioned natural language-to-SQL query method based on execution result verification, which includes the following steps:

[0093] Inputting the natural language question into a Text-to-SQL dataset, and selecting several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset;

[0094] Input the examples and their corresponding SQL queries and the natural language question into the LLM, and the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries;

[0095] Execute the candidate SQL query, input the obtained execution result, the natural language question, and the candidate SQL query into the trained verification model to determine the correctness of the execution result and give a correct probability of the execution result;

[0096] The correct probability of the candidate SQL query is calculated according to the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, the correct probabilities of the candidate SQL queries are arranged in descending order, and the first candidate SQL query is output.

[0097] Of course, the processor of the server provided in the embodiment of the present invention is not limited to executing the method operations described above, but can also execute related operations in the natural language to SQL query method based on execution result verification provided in any embodiment of the present invention.

[0098] The memory 350 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 350 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 350 may further include a memory remotely arranged relative to one or more CPUs 340, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] The input / output 320 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The input / output 320 may also include a display device such as a display screen.

[0100] The embodiment of the present invention also provides a non-temporary computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor, the natural language to SQL query method based on execution result verification described in the above embodiment is implemented. The computer-readable storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0101] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0102] The program code contained on the storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0103] In addition, other specific working processes of a non-temporary computer-readable storage medium refer to the description of the above-mentioned natural language to SQL query method embodiment based on execution result verification, and will not be repeated here.

[0104] Based on the technical solutions provided in the above embodiments, a natural language to SQL query method, system and storage medium based on execution result verification use examples to generate candidate SQL queries. Similar question and SQL query pairs are obtained as examples in the Text-to-SQL dataset through natural language questions. The questions, database schema and examples are provided to the large language model, which generates multiple SQL queries that can answer the provided questions based on the examples. This method can ensure the diversity of generated SQL queries and increase the probability of selecting the correct SQL query through verification.

[0105] Verify the correctness of the generated SQL query through the execution results. Use the Text-to-SQL dataset to train a verification model. The verification model judges the correctness of the execution result based on the natural language question, the generated SQL query and its execution result in the database, and outputs the probability of the execution result being correct. Using the execution results, it is easy to find that the generated SQL query has an incorrect execution result due to syntax errors. For complex questions and SQL queries, the execution results can more intuitively determine the correctness of the SQL query.

[0106] Reorder the candidate SQL queries according to the correct probability of the SQL query. Calculate the correct probability of the SQL query by combining the generation probability of the SQL query generated by the large language model and the correct probability of the execution result verification. Rearrange the candidate SQL queries in order from large to small correct probability, and output the SQL query with the highest correct probability. Adjusting the order of the candidate SQL queries according to the correctness of the execution results helps to improve the correctness of the SQL query finally output. In summary, the beneficial effects of the method of the present invention include:

[0107] At present, most Text-to-SQL methods directly output the generated SQL query and verify the correctness of the SQL query. The Text-to-SQL method for verifying the generated SQL query mainly judges the correctness of the SQL query at the semantic level through natural language questions and database patterns, and it is difficult to find errors in complex questions and complex SQL queries. The present invention:

[0108] A method for generating SQL queries by using natural language questions or examples is proposed. By generating multiple candidate SQL queries, the diversity of generated SQL queries is ensured and the possibility of correct SQL queries being present in the generated SQL queries is increased.

[0109] This paper proposes a method to verify SQL queries through the execution results of SQL queries. The execution results can more easily find the syntax errors of SQL queries, more intuitively judge the correctness of SQL queries, and help to find complex problems and errors in complex SQL queries.

[0110] A candidate SQL query re-ranking method is proposed. The candidate SQL queries are re-ranked according to the generation probability of the SQL query and the correct probability of its execution result, thereby increasing the possibility of selecting the correct SQL query and thus improving the accuracy of the Text-to-SQL method.

[0111] In this document, the terms "comprises," "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that includes a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such step or method.

[0112] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A natural language to SQL query method based on execution result verification, characterized in that: The method comprises the following steps: Inputting the natural language question into a Text-to-SQL dataset, and selecting several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset; Input the examples and their corresponding SQL queries and the natural language question into the LLM, and the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries; Execute the candidate SQL query, input the obtained execution result, the natural language question, and the candidate SQL query into the trained verification model to determine the correctness of the execution result and give a correct probability of the execution result; The correct probability of the candidate SQL query is calculated according to the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, the correct probabilities of the candidate SQL queries are arranged in descending order, and the first candidate SQL query is output.

2. The natural language to SQL query method based on execution result verification according to claim 1, characterized in that: The natural language question is input into a Text-to-SQL dataset, and several examples similar to the natural language question and their corresponding SQL queries are selected from the Text-to-SQL dataset. The specific implementation process includes: using word embedding technology to convert the natural language question and each natural language question in the Text-to-SQL dataset into numerical vectors respectively; based on the numerical vector, the distance between the natural language question and each natural language question in the Text-to-SQL dataset is calculated by Euclidean distance, and the first N natural language questions in the Text-to-SQL dataset are selected as examples after arranging in ascending order.

3. The natural language to SQL query method based on execution result verification according to claim 1, characterized in that: The training process of the verification model includes: Build a dataset for training and testing the model: execute the SQL queries in the Text-to-SQL dataset in the database to obtain the execution results, and mark whether the execution results are correct; The verification model is trained using the constructed data set: natural language questions, SQL queries and their execution results are used as inputs to the verification model, and the verification model outputs the correct probability of the execution result.

4. According to claim 1, the natural language to SQL query method based on execution result verification is characterized in that: The correct probability of the candidate SQL query is calculated according to the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, specifically including: multiplying the generation probability of each candidate SQL query in the LLM with the correct probability of the execution result obtained by the candidate SQL query in the verification model to obtain the correct probability of the candidate SQL query.

5. A natural language to SQL query system based on execution result verification, characterized in that: The system comprises: a candidate SQL query generation module, configured to input the natural language question into a Text-to-SQL dataset, and select several examples similar to the natural language question and their corresponding SQL queries from the Text-to-SQL dataset; and Input the examples and their corresponding SQL queries and the natural language question into the LLM, and the LLM outputs candidate SQL queries and reasons for generating the candidate SQL queries; An SQL query verification module is used to execute the candidate SQL query, input the obtained execution result, the natural language question, and the candidate SQL query into a trained verification model to determine the correctness of the execution result and give a correct probability of the execution result; The candidate SQL query rearrangement module is used to calculate the correct probability of the candidate SQL query according to the probability of generating the candidate SQL query by the LLM and the correct probability of the execution result, arrange the correct probabilities of the candidate SQL queries in descending order and output the first candidate SQL query.

6. The natural language to SQL query system based on execution result verification according to claim 5, characterized in that: In the candidate SQL query generation module, the natural language question is input into a Text-to-SQL data set, and several examples similar to the natural language question and their corresponding SQL queries are selected from the Text-to-SQL data set. The specific implementation process includes: using word embedding technology to convert the natural language question and each natural language question in the Text-to-SQL data set into numerical vectors respectively; based on the numerical vectors, the distance between the natural language question and each natural language question in the Text-to-SQL data set is calculated by Euclidean distance, and the first N natural language questions in the Text-to-SQL data set are selected as examples after arranging in ascending order. .

7. The natural language to SQL query system based on execution result verification according to claim 5, characterized in that: In the SQL query verification module, the training process of the verification model includes: Build a dataset for training and testing the model: execute the SQL queries in the Text-to-SQL dataset in the database to obtain the execution results, and mark whether the execution results are correct; The verification model is trained using the constructed data set: natural language questions, SQL queries and their execution results are used as inputs to the verification model, and the verification model outputs the correct probability of the execution result.

8. The natural language to SQL query system based on execution result verification according to claim 5, characterized in that: In the candidate SQL query rearrangement module, the correct probability of the candidate SQL query is calculated based on the probability of the LLM generating the candidate SQL query and the correct probability of the execution result, specifically including: multiplying the generation probability of each candidate SQL query in the LLM with the correct probability of the execution result obtained by the candidate SQL query in the verification model to obtain the correct probability of the candidate SQL query.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the natural language to SQL query method based on execution result verification as described in any one of claims 1 to 4 are implemented.

10. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the natural language to SQL query method based on execution result verification as described in any one of claims 1 to 4 are implemented.

Images