System and method for improving reflection ability of NL2SQL model based on SQL grammar check

Abstract

The invention discloses a system and method for improving the reflection ability of an NL2SQL model based on SQL grammar check, and relates to the technical field of NL2SQL model application, the system comprises an Actor module, the NL2SQL model, an Evaluator module, a Self-Reflection module, a database and the like, the system generates an SQL query statement by receiving a natural language query of a user, the generated SQL query statement is subjected to grammar check and grammar verification, and the generated SQL query statement is subjected to the grammar check and the grammar verification to improve the reflection ability of the NL2SQL model. And reflecting the grammar verification error information to obtain a correct SQL query statement. According to the method, the answering capability of the NL2SQL model is improved based on a reflection mechanism of data SQL grammar check, so that the NL2SQL model has self-reflection and correction capabilities, the query generated by the NL2SQL model is more accurate, efficient and reliable, and the accuracy and reliability of the NL2SQL model are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of database interactive application using an NL2SQL model, and in particular to a system and method for improving the reflection capability of an NL2SQL model based on SQL syntax checking. Background Art

[0002] Interacting with a database usually requires a certain amount of technical expertise, which makes it difficult for many people to access data. NL2SQL (a function that converts natural language into structured query language statements) changes this situation. With NL2SQL, users can interact with the database directly using natural language. The NL2SQL model generates corresponding database query statements based on the user's natural language, greatly reducing the operational threshold for database queries.

[0003] However, the structure of real enterprise application databases is more complex, the analysis logic is more complex, and the accuracy and corresponding performance requirements are high. The prediction of deep learning AI models has its own confidence problem and cannot ensure absolute reliability. This still exists in large language models, especially in NL2SQL tasks. Natural language expressions are ambiguous, and SQL is a precise programming language. Therefore, in actual applications, there may be situations where it is impossible to understand or misunderstand. For example, "Who is the most powerful sales this month?", then does AI understand it as the largest number of orders, or the largest order amount? The uncertainty of the output is also the biggest obstacle to the application of large models in key enterprise systems.

[0004] Reflection is a prompting strategy used to improve the quality and success of agents and similar AI systems. It involves prompting LLMs to reflect on and criticize their past actions, sometimes in conjunction with other external information, such as observational analysis by tools.

[0005] For example: The invention application with application number 202410072558.4 mentions a multi-layer semantic-aware enhanced NL2SQL model and method. Using the solution of this application, many non-expert users can also use natural language to query data in the database, saving the user's learning cost of learning SQL query statement-related knowledge, improving the accuracy of the model, and reducing the complexity of the model; the invention application with application number 202411009128.4 mentions a self-enhanced fine-tuning method and device for a large NL2SQL language model. Through this application solution, the training cycle can be shortened, the training cost can be reduced, and the training level can be improved.

[0006] Although the above scheme improves the NL2SQL model application, it lacks a reflection mechanism, and there is still a problem that NL2SQL cannot accurately understand the SQL query statement, or misunderstands it, resulting in a decrease in the accuracy and reliability of the NL2SQL application. Summary of the invention

[0007] The purpose of the present invention is to provide a system and method for improving the reflection ability of the NL2SQL model based on SQL syntax checking, adopting a reflection mechanism for the NL2SQL model, and improving the accuracy and reliability of the NL2SQL model application.

[0008] The embodiment of the present invention provides a system and method for improving the introspection capability of an NL2SQL model based on SQL grammar checking.

[0009] A first aspect: A system for improving the reflection capability of NL2SQL model based on database SQL syntax checking, comprising:

[0010] Actor module, which receives the user's natural language query and obtains the prompt solution;

[0011] NL2SQL model, generates SQL query statements;

[0012] The Evaluator module performs syntax checking on the generated SQL query statements;

[0013] The database uses the executor to perform syntax verification on SQL query statements;

[0014] The Self-Reflection module reflects on syntax validation error messages, helps obtain the correct SQL query statements, and returns them to the user.

[0015] The second aspect: A method for improving the reflection ability of the NL2SQL model by checking the database SQL syntax based on the above system, including:

[0016] S1. Receive the user's natural language query and obtain the prompt solution;

[0017] S2, generating SQL query statements based on the user's natural language query prompt scheme;

[0018] S3, perform syntax check on the generated SQL query statement;

[0019] S4, performing syntax verification on the SQL query statement that has passed the syntax check;

[0020] S5. Reflect on the syntax verification error information, obtain the correct SQL query statement, and return it to the user.

[0021] Furthermore, the S3 performs syntax checking on the generated SQL query statement using the SQLFluff syntax library.

[0022] Furthermore, the S3 comprises the steps of:

[0023] S31, using the SQLFluff grammar library to perform grammar check on the generated SQL query statement;

[0024] S32, if the grammar check is successful, then perform grammar verification;

[0025] S33. If the syntax check fails, the SQL query statement is resent to the NL2SQL model, and syntax correction is performed iteratively until the syntax check succeeds.

[0026] Further, the S5 comprises the steps of:

[0027] S51, use the Self-Reflection module to parse and verify the error information, and correct the SQL query statement;

[0028] S52. Send the corrected SQL query statement to the NL2SQL model to regenerate a new SQL query statement:

[0029] S53. Use the database to perform executor syntax verification on the new SQL query statement again.

[0030] Furthermore, it also includes:

[0031] S54. If the syntax verification is successful, it is returned to the user; if the syntax verification is unsuccessful, S51 to S53 are iterated no more than three times.

[0032] Furthermore, the Prompt scheme in S1 includes six elements: instruction (Instruction), data structure (TableSchema), reference sample (Sample), other prompts (Tips) / constraints (Constraint), domain knowledge (Knowledge) and user question (Question).

[0033] A third aspect: An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method provided in the second aspect when executing the program.

[0034] A fourth aspect: A non-transitory computer-readable storage medium having a computer program stored thereon, which implements the steps of the method provided in the second aspect when executed by a processor.

[0035] Beneficial effects of the present invention:

[0036] 1. The method of the present invention for improving the answering ability of the NL2SQL model based on the reflection mechanism of data SQL syntax checking uses the NL2SQL model optimized with prompt words to generate corresponding SQL query statements according to natural language questions, and uses the reflection mechanism to perform syntax checking on the generated SQL query statements. The NL2SQL model is equipped with self-reflection and correction capabilities, so that the queries it generates are more accurate, efficient and reliable, and the accuracy and reliability are significantly improved.

[0037] 2. Implement syntax checking through SQL syntax analysis tools or custom rules, and verify the SQL execution results in combination with database SQL syntax checking methods to obtain the final legality of the generated SQL. By adding a reflection mechanism for SQL syntax checking through database syntax management, the accuracy of the NL2SQL model's answers can be effectively improved. This reflection mechanism can provide reliable data retrieval and improve the user's ability to correctly execute natural language queries, making it a valuable tool for seeking to optimize the organization of data query processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a principle flow chart of the system for improving the reflection capability of the NL2SQL model based on database SQL syntax checking of the present invention;

[0039] Figure 2 It is a flow chart of a method for improving the reflection capability of the NL2SQL model based on database SQL syntax checking of the present invention;

[0040] Figure 3 This is a sample diagram of the Prompt solution of the present invention;

[0041] Figure 4 A core code graph for syntax checking of the present invention.

[0042] Figure 5 This is a sample diagram of the Prompt scheme used in the reflection mechanism of the present invention;

[0043] Figure 6 This is the core code diagram for syntax verification of the present invention.

[0044] Figure 7 It is the working flow chart of the system of the present invention;

[0045] Figure 8 It is a schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0046] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar symbols throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0047] The existing NL2SQL model has problems with task execution because natural language expressions are ambiguous and SQL is a precise programming language. Therefore, in practical applications, the system may not understand or may misunderstand the model.

[0048] In view of the above problems, the present invention provides a system for improving the reflection ability of NL2SQL model based on database SQL syntax checking. Figure 1 The principle flow chart of the system provided by the embodiment of the present invention includes: Actor (action) module, NL2SQL model, Evaluator (check) module, Self-Reflection (reflection) module and database, etc.

[0049] Among them, the Actor module is used to receive the user's natural language query and obtain the Prompt solution; the NL2SQL model generates SQL query statements based on the Prompt solution; the Evaluator module is used to perform syntax checking on the generated SQL query statements; the database uses the executor to perform syntax verification on the SQL query statements after syntax checking; the Self-Reflection module reflects on the syntax verification error information, corrects the SQL query statements, and helps obtain the correct SQL query statements. The system obtains the correct SQL query statements and returns them to the user.

[0050] The Actor module is mainly used to accept natural language queries from users. However, the LLM model's understanding of natural language queries from users is closely related to whether the prompt word (Prompt) is appropriate. Prompt can guide LLM to generate text that meets the expected content, and Prompt can control the direction and content of the results generated by LLM.

[0051] After a lot of practice, the Actor module of the present invention adopts a relatively general Prompt solution to process the natural language query received from the user.

[0052] The prompt solution includes six elements: instruction, data structure (Table Schema), reference sample (Sample), other prompts (Tips) / constraints (Constraint), domain knowledge (Knowledge) and user questions (Question).

[0053] Instruction: For example, "You are an expert in SQL generation. Please refer to the following table structure and directly output the SQL query statement without any unnecessary explanation."

[0054] Data structure (Table Schema): Similar to the "vocabulary" in language translation, that is, the database table structure to be used. Since the LLM large model cannot directly access the database, the data structure needs to be assembled into the prompt, usually including the table name, column name, column type, column meaning, primary and foreign key information.

[0055] Reference example: Figure 7 As shown, this is an optional option and a common technique for prompting engineering, that is, to guide the large model to generate a reference example of SQL for this time.

[0056] Other Tips / Constraints: These are other instructions that are considered necessary; for example, requiring that expressions not be allowed in the generated SQL, or requiring that column names must be in the form of "table.column".

[0057] Domain knowledge: This is an optional item. In some specific questions, such as "Who is the best salesperson this month?", you need to tell the model whether "best" means "the most sales orders" or "the most sales amount."

[0058] User questions: questions expressed in natural language, such as "calculate the average order amount last month".

[0059] Based on the above design, the Actor module of the present invention uses a self-developed Prompt solution to meet the needs of users or enterprises. A specific Prompt solution example, taking an actual scenario as an example, the partial Prompt implementation after Format is as follows Figure 3 shown.

[0060] The Evaluator module performs syntax checking on the generated SQL query statements. The Evaluator (checking) module may use the SQLFluff syntax library. The SQLFluff syntax library is a library for SQL syntax checking, and is particularly suitable for static analysis and formatting of SQL codes. The Evaluator module of the present invention uses the SQLFluff syntax library by default to perform syntax checking on the SQL query statements generated by the NL2SQL model. The syntax checking core code is as follows: Figure 4 shown.

[0061] The Self-Reflection module uses a retry mechanism for errors received from the executor. When the generated SQL query statement causes an error, the Self-Reflection module engine will use an agent with a predefined template to correct the query, forming a reflection mechanism. For example, the prompt used in some reflection mechanisms is as follows Figure 5 shown.

[0062] By using advanced LLM and carefully designed prompts, the Self-Reflection module can reflect on syntax validation error messages, correct SQL query statements, and generate more accurate and context-sensitive SQL query statements.

[0063] After the Self-Reflection module corrects the SQL query statement, it is sent to the specified database executor for syntax verification. For example, the database Mysql is used as follows: Send to executor for syntax verification. The core code is as follows: Figure 6 shown.

[0064] When the database executor returns that the execution is correct, it will return the SQL query statement directly to the user. Otherwise, it will combine the above prompt, the user question and the returned error information, and then correct the SQL query again to obtain the corrected SQL query statement. After that, it will continue to iterate according to this mode, but the number of iterations generally does not exceed three times.

[0065] The present invention also discloses a method for improving the answering ability of the NL2SQL model based on the reflection mechanism based on data SQL syntax checking of the above system, using the NL2SQL model optimized by prompt words to generate corresponding SQL query statements according to natural language questions, and using the reflection mechanism to perform syntax verification on the generated SQL query statements.

[0066] Platform workflow Figure 2 As shown, the steps include:

[0067] Step 1: The Actor module receives the user's natural language query, such as "query the name and age of all employees".

[0068] Step 2: The NL2SQL model generates SQL query statements, for example: SELECT name, age FROM employees.

[0069] Step 3: Evaluator uses the sqlfluff grammar library to perform a grammar check on the generated SQL query statement. If the grammar check succeeds, it proceeds to step 4. Otherwise, it will be resent to the NL2SQL model for grammar correction, and iterates continuously until the set conditions are met.

[0070] Step 4: Send the SQL statement corrected in step 3 to the database for syntax verification using the executor. If the database feedback indicates that the query syntax is correct, it is returned to the user.

[0071] Step 5: If the database returns a syntax error, such as "table 'employees' does not exist", the reflection module parses the error message and adjusts the SQL query statement. Use the NL2SQL model to regenerate the corrected SQL query statement: for example: SELECT name, age FROM staff.

[0072] Step 6: The database verifies the newly generated SQL query statement and confirms that its syntax is correct. For the SQL query statement that fails the syntax verification again, the Self-Reflection module is returned to continue the syntax parsing verification error information, and then iterates continuously according to this mode, but generally the number of iterations does not exceed three times.

[0073] Step 7: The system returns the final SQL query statement to the user.

[0074] The method of adding the reflection mechanism of the present invention is compared with the existing method by using a question for comparison:

[0075] User enters a natural language query: "Determine the quarterly trend of student A's physics grades in 2023"

[0076] The SQL query statement generated before using the reflection mechanism of the present invention is:

[0077] SELECT SUM(physics_score)AS score FROM student_score

[0078] WHERE year=2023AND class_Name='Physics'AND student_id='A'GROUP BYquarter_number';

[0079] Problems: First, the query above groups by quarter_number but does not select the quarterly breakdown, which may result in incomplete results. Second, student_id is used instead of a more logical identifier like student_name. Third, SUM(physics_score) does not handle possible division by zero.

[0080] Average latency: 5 seconds; The previous setup for the reflection mechanism used a combination of GPT-3.5 with prompt engineering and 5+ small sample queries per user prompt.

[0081] The SQL query statement generated by the reflection mechanism of the present invention is:

[0082] SELECTyear,quarter_number,COALESCE(SUM(NULLIF(physics_score,0)),1)ASscore FROM student_score

[0083] WHERE class_Name='Physics'AND student_name='A'AND year=2023GROUP BY quarter_number, year ORDER BY quarter_number ASC;

[0084] The improvements include: first, the query includes quarter_number and provides the necessary quarter details; second, the student_name field is used; third, more precise identifiers are provided for indicators, the COALESCE(SUM(NULLIF(physics_score,0)),1) function handles potential division by zero errors, and the results are sorted by quarter number to reflect quarterly trends.

[0085] Average latency: 7 seconds. The reflection mechanism takes time and is related to the NL2SQL model used. Although some extra time is sacrificed, the calculation is used to obtain better quality output, which is more important for knowledge-intensive tasks where response quality is more important than speed.

[0086] The above comparison shows the accuracy and reliability results before and after using the reflection mechanism based on the benchmark of the production workload:

[0087] Without using this reflection mechanism: Accuracy: 65% Reliability: 60%.

[0088] Using this reflection mechanism: Accuracy: 92% Reliability: 90%.

[0089] The above results are from an internal benchmarking tool that executes each prompt 100 times with separate identifiers to eliminate the effects of caching. The suite measures accuracy by comparing the returned response to a benchmark response, and reliability by measuring how often similar responses are returned.

[0090] The comparison results clearly show the advantages of the reflection mechanism of the present invention in converting natural language queries into precise SQL queries. After using the reflection mechanism, the overall performance is improved by 30%.

[0091] The present invention also provides an electronic device, Figure 8A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Figure 8 As shown, the electronic device may include: a processor, a communications interface, a memory, and a communications bus, wherein the processor, the communications interface, and the memory communicate with each other via the communications bus. The processor may call the logic instructions in the memory, for example, to execute the following method:

[0092] S1. Receive the user's natural language query and obtain the prompt solution;

[0093] S2, generating SQL query statements based on the user's natural language query prompt scheme;

[0094] S3, perform syntax check on the generated SQL query statement;

[0095] S4, performing syntax verification on the SQL query statement that has passed the syntax check;

[0096] S5. Reflect on the syntax verification error information and obtain the correct SQL query statement;

[0097] S6. Return the SQL query statement based on the correct SQL query statement.

[0098] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0099] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

[0100] S1. Receive the user's natural language query and obtain the prompt solution;

[0101] S2, generating SQL query statements based on the user's natural language query prompt scheme;

[0102] S3, perform syntax check on the generated SQL query statement;

[0103] S4, performing syntax verification on the SQL query statement that has passed the syntax check;

[0104] S5. Reflect on the syntax verification error information and obtain the correct SQL query statement;

[0105] S6. Return the SQL query statement based on the correct SQL query statement.

[0106] The system embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A system for improving the introspection capability of NL2SQL model based on SQL syntax checking, characterized in that: include: Actor module, which receives the user's natural language query and obtains the prompt solution; NL2SQL model, generates SQL query statements; The Evaluator module performs syntax checking on the generated SQL query statements; The database uses the executor to perform syntax verification on SQL query statements; The Self-Reflection module reflects on syntax validation error messages, helps obtain the correct SQL query statements, and returns them to the user.

2. A method based on the system of claim 1, characterized in that: Includes steps: S1. Receive the user's natural language query and obtain the prompt solution; S2, generating SQL query statements based on the user's natural language query prompt scheme; S3, perform syntax check on the generated SQL query statement; S4, performing syntax verification on the SQL query statement that has passed the syntax check; S5. Reflect on the syntax verification error information, obtain the correct SQL query statement, and return it to the user.

3. The method according to claim 2, characterized in that The generated SQL query statement is syntax-checked in the S3 using the SQLFluff syntax library.

4. The method according to claim 3, characterized in that: The S3 comprises the steps of: S31, using the SQLFluff grammar library to perform grammar check on the generated SQL query statement; S32, if the grammar check is successful, then perform grammar verification; S33. If the syntax check fails, the SQL query statement is resent to the NL2SQL model, and syntax correction is performed iteratively until the syntax check succeeds.

5. The method according to claim 2, characterized in that: The S5 comprises the steps of: S51, use the Self-Reflection module to parse and verify the error information, and correct the SQL query statement; S52. Send the corrected SQL query statement to the NL2SQL model to regenerate a new SQL query statement: S53. Use the database to perform executor syntax verification on the new SQL query statement again.

6. The method according to claim 5, characterized in that Also includes: S54. If the syntax verification is successful, it is returned to the user; If the syntax verification fails, S51 to S53 are iteratively executed no more than three times.

7. The method according to claim 2, characterized in that The Prompt scheme in S1 includes six elements: instruction, data structure (Table Schema), reference sample (Sample), other prompts (Tips) / constraints (Constraint), domain knowledge (Knowledge) and user question (Question).

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 2 to 7 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 2 to 7 are implemented.

Images