SQL statement generation method and device based on retrieval enhancement type generation

Abstract

The invention relates to an SQL statement generation method and device based on retrieval enhancement type generation. The method comprises the following steps: acquiring database information of various business scenes; according to the table description information, generating a table description fragment of each business scene by using a large language model, vectorizing the table description fragments, and storing the vectorized table description fragments into a vector database; according to the database information, generating various types of SQL query statements by using the large language model, regenerating a natural language query statement by using the SQL query statements and the large language model, forming a natural language-SQL statement data pair, vectorizing the natural language-SQL statement data pair, and storing the vectorized natural language-SQL statement data pair into a vector database; receiving a query statement input by a user, vectorizing the query statement, and recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval; and generating an SQL query statement according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information. By adopting the method, the accuracy of SQL statement generation can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of SQL statement generation, and in particular to a method and device for generating negative cases in bidding documents based on a large language model. Background Art

[0002] In the field of natural language processing, NL2SQL (natural language to SQL statement conversion) has always been a very popular and difficult research field. Before the emergence of LLM (large language model), the traditional approach was to directly train the model of natural language to SQL statement based on seq2seq model, or to use two seq2seq models, first use one model to convert natural language into a custom intermediate language model (schema), and then use the second model to convert such intermediate results into SQL statements.

[0003] With the development of large models (LLM), the NL2SQL task has made new breakthroughs. With the support of semantic understanding + code generation capabilities based on GPT4, it has quickly surpassed traditional algorithms. The main approach of these solutions is to use the semantic understanding ability of large models in conjunction with database table information to directly generate SQL statements. However, large models cannot adapt to databases in different scenarios in a targeted manner, so database information (including table names, column names, etc.) needs to be input into the large model as context. At this time, if the database information is too long, the context of the large model cannot be accommodated, which will greatly reduce the accuracy of the generated results. If you only choose to retrain the large model based on database information (post-pretrain), the second problem will arise. Every update of the database (addition and subtraction of tables, addition and subtraction of columns) requires retraining of the large model. Such computing power and time costs are unacceptable. Summary of the invention

[0004] Based on this, it is necessary to provide a method and device for generating negative cases in bidding documents based on a large language model to address the above technical problems.

[0005] A method for generating SQL statements based on retrieval-enhanced generation, the method comprising:

[0006] Obtaining database information for various business scenarios; the database information includes table description information;

[0007] Generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database;

[0008] Generate various types of SQL query statements using a large language model according to the database information, use the SQL query statements to generate natural language query statements again using the large language model, combine the SQL query statements and the natural language query statements into a natural language-SQL statement data pair, and store them in a vector database after vectorization; the vector database stores the database information, table description fragments, and the correspondence between the natural language-SQL statement data pairs;

[0009] receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval;

[0010] An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

[0011] In one embodiment, the table description information includes: table name, table information introduction, column name, column information introduction, specific element examples and table foreign key information.

[0012] In one embodiment, each type of SQL query statement includes a simple query and a complex query.

[0013] In one of the embodiments, syntax verification and result verification are performed on the generated SQL query statement, and the SQL statement or query result is output as required.

[0014] A device for generating SQL statements based on retrieval-enhanced generation, the device comprising:

[0015] A database information acquisition module is used to acquire database information for various business scenarios; the database information includes table description information;

[0016] A table description information acquisition module, used to generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database;

[0017] A data pair construction module, used to generate various types of SQL query statements using a large language model according to the database information, and to generate natural language query statements again using the SQL query statements and the large language model, and to form a natural language-SQL statement data pair with the SQL query statement and the natural language query statement, and to vectorize and store the natural language-SQL statement data pair in a vector database; the vector database stores the database information, the table description fragment, and the correspondence between the natural language-SQL statement data pair;

[0018] receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval;

[0019] An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

[0020] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0021] Obtaining database information for various business scenarios; the database information includes table description information;

[0022] Generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database;

[0023] Generate various types of SQL query statements using a large language model according to the database information, use the SQL query statements to generate natural language query statements again using the large language model, combine the SQL query statements and the natural language query statements into a natural language-SQL statement data pair, and store them in a vector database after vectorization; the vector database stores the database information, table description fragments, and the correspondence between the natural language-SQL statement data pairs;

[0024] receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval;

[0025] An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

[0026] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0027] Obtaining database information for various business scenarios; the database information includes table description information;

[0028] Generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database;

[0029] Generate various types of SQL query statements using a large language model according to the database information, use the SQL query statements to generate natural language query statements again using the large language model, combine the SQL query statements and the natural language query statements into a natural language-SQL statement data pair, and store them in a vector database after vectorization; the vector database stores the database information, table description fragments, and the correspondence between the natural language-SQL statement data pairs;

[0030] receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval;

[0031] An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

[0032] The above-mentioned SQL statement generation method, device, computer equipment and storage medium based on retrieval-enhanced generation collects database information of various business scenarios, generates table description fragments of each business scenario based on the database information using a large language model, and vectorizes the table description fragments and stores them in a vector database. Then, based on the database information, a SQL query statement is generated using a large language model, and then a natural language query statement is reversely generated through the SQL query statement to obtain a natural language-SQL statement data pair. In the present invention, the natural language-SQL statement data pair can play a demonstration role when the large language model performs the generation task, thereby improving the accuracy of SQL statement generation. Finally, the user inputs the query statement, recalls the table description fragment and the natural language-SQL statement data pair in the vector database, and then generates the SQL statement using the large language model, which can solve the semantic gap between natural language and database description information, thereby improving the accuracy of SQL statement generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flowchart of a method for generating a SQL statement based on retrieval-enhanced generation in one embodiment;

[0034] Figure 2 A framework diagram of a method for generating a SQL statement based on retrieval-enhanced generation in another embodiment;

[0035] Figure 3 An internal structural diagram of a computer device in one embodiment. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0037] In one embodiment, Figure 1 As shown, a method for generating SQL statements based on retrieval-enhanced generation is provided, comprising the following steps:

[0038] Step 102, obtaining database information of various business scenarios.

[0039] Database information includes table description information.

[0040] It is worth noting that in different business scenarios, corresponding to different databases and data table fields, when generating SQL statements, different SQL statements are generated for different tables and fields. Therefore, once fields are added or deleted, the original SQL statements need to be adjusted.

[0041] In this step, different database information needs to be collected for different business scenarios, and the database information includes table description information and field description information.

[0042] Step 104: Generate table description segments for each business scenario using a large language model based on the table description information, and vectorize the table description segments and store them in a vector database.

[0043] The length of the table description text varies in different business scenarios. The longest table description may be tens of thousands of words long. Directly inputting the table description information into the big model for SQL generation will cause the model to be unable to tolerate the context, resulting in a decrease in generation accuracy. In this step, the big model is used to generate table description fragments for the table description information of each business scenario, and the accuracy of the generated table description fragments is improved by continuously optimizing the big model.

[0044] Step 106, based on the database information, use the large language model to generate various types of SQL query statements, use the SQL query statements to use the large language model to generate natural language query statements again, combine the SQL query statements and the natural language query statements into a natural language-SQL statement data pair and store them in a vector database after vectorization.

[0045] The vector database stores the correspondence between database information, table description fragments, and natural language-SQL statement data pairs.

[0046] In this step, we first traverse and generate SQL query statements for the existing database, and then use the database information to generate SQL statements. After screening and deduplication, we use the SQL statements to generate natural language, thus breaking the understanding gap between database information and natural language.

[0047] Step 108, receiving the query statement input by the user, vectorizing the query statement, and retrieving the natural language query statement and table description fragments in the vector database through similarity retrieval.

[0048] Since the understanding between database information and natural language is enhanced in step 106, after receiving the user's query statement, natural language-SQL statement data pairs and table description fragments can be directly recalled through semantic similarity.

[0049] Step 110: Generate an SQL query statement according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment, and the database information.

[0050] In the above-mentioned SQL statement generation method based on retrieval-enhanced generation, by collecting database information of various business scenarios, based on the database information, a large language model is used to generate table description fragments for each business scenario, and the table description fragments are vectorized and stored in the vector database. Then, based on the database information, a SQL query statement is generated using a large language model, and then a natural language query statement is reversely generated through the SQL query statement to obtain a natural language-SQL statement data pair. In the present invention, the natural language-SQL statement data pair can play a demonstration role when the large language model performs the generation task, thereby improving the accuracy of SQL statement generation. Finally, the user inputs the query statement, and the table description fragment and the natural language-SQL statement data pair are recalled in the vector database, and then the large language model is used to generate the SQL statement, which can solve the semantic gap between the natural language and the database description information, thereby improving the accuracy of SQL statement generation.

[0051] In one embodiment, the table description information includes: table name, table information introduction, column name, column information introduction, specific element examples and table foreign key information.

[0052] In one embodiment, each type of SQL query statement includes a simple query and a complex query.

[0053] In one embodiment, if Figure 2 As shown, the generated SQL query statement is syntax-checked and result-checked, and the SQL statement or query result is output as required.

[0054] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0055] In one embodiment, a device for generating SQL statements based on retrieval-enhanced generation is provided, comprising:

[0056] A database information acquisition module is used to acquire database information for various business scenarios; the database information includes table description information;

[0057] A table description information acquisition module, used to generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database;

[0058] A data pair construction module, used to generate various types of SQL query statements using a large language model according to the database information, and to generate natural language query statements again using the SQL query statements and the large language model, and to form a natural language-SQL statement data pair with the SQL query statement and the natural language query statement, and to vectorize and store the natural language-SQL statement data pair in a vector database; the vector database stores the database information, the table description fragment, and the correspondence between the natural language-SQL statement data pair;

[0059] receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval;

[0060] An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

[0061] In one embodiment, the table description information includes: table name, table information introduction, column name, column information introduction, specific element examples and table foreign key information.

[0062] In one embodiment, each type of SQL query statement includes a simple query and a complex query.

[0063] In one of the embodiments, syntax verification and result verification are performed on the generated SQL query statement, and the SQL statement or query result is output as required.

[0064] For the specific definition of the SQL statement generation device based on retrieval enhancement, please refer to the definition of the SQL statement generation method based on retrieval enhancement above, which will not be repeated here. Each module in the above-mentioned SQL statement generation device based on retrieval enhancement can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0065] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for generating SQL statements based on retrieval-enhanced generation is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0066] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0067] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0068] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0069] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0070] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0071] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for generating SQL statements based on retrieval-enhanced generation, characterized in that: The method comprises: Obtaining database information for various business scenarios; the database information includes table description information; Generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database; Generate various types of SQL query statements using a large language model according to the database information, use the SQL query statements to generate natural language query statements again using the large language model, combine the SQL query statements and the natural language query statements into a natural language-SQL statement data pair, and store them in a vector database after vectorization; the vector database stores the database information, table description fragments, and the correspondence between the natural language-SQL statement data pairs; receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval; An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

2. The method according to claim 1, characterized in that: The table description information includes: table name, table information introduction, column name, column information introduction, specific element examples and table foreign key information.

3. The method according to claim 1, characterized in that Various types of SQL query statements include simple queries and complex queries.

4. The method according to claim 1, characterized in that: The method further comprises: Perform syntax verification and result verification on the generated SQL query statements, and output SQL statements or query results as required.

5. A device for generating SQL statements based on retrieval-enhanced generation, characterized in that: The device comprises: A database information acquisition module is used to acquire database information for various business scenarios; the database information includes table description information; A table description information acquisition module, used to generate a table description segment for each business scenario using a large language model according to the table description information, and vectorize the table description segment and store it in a vector database; A data pair construction module, used to generate various types of SQL query statements using a large language model according to the database information, and to generate natural language query statements again using the SQL query statements and the large language model, and to form a natural language-SQL statement data pair with the SQL query statement and the natural language query statement, and to vectorize and store the natural language-SQL statement data pair in a vector database; the vector database stores the database information, the table description fragment, and the correspondence between the natural language-SQL statement data pair; receiving a query statement input by a user, vectorizing the query statement, and then recalling the natural language query statement and the table description fragment in the vector database through similarity retrieval; An SQL query statement is generated according to the natural language-SQL statement data pair corresponding to the natural language query statement, the table description fragment and the database information.

6. The device according to claim 5, characterized in that The table description information includes: table name, table information introduction, column name, column information introduction, specific element examples and table foreign key information.

7. The device according to claim 5, characterized in that Various types of SQL query statements include simple queries and complex queries.

8. The device according to claim 5, characterized in that Perform syntax verification and result verification on the generated SQL query statements, and output SQL statements or query results as required.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images