Reinforcement learning-based natural language to SQL model training method and device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BANK OF COMMUNICATIONS
- Filing Date
- 2026-04-08
- Publication Date
- 2026-07-10
AI Technical Summary
Existing natural language to SQL models struggle to accurately retrieve table structures, understand field meanings, handle complex logical queries, and adapt to dynamic table updates when faced with complex enterprise databases and dynamic data environments, resulting in insufficient robustness.
We employ a reinforcement learning-based approach, combining external tools and an SQL validator. By verifying tool call trajectories and results, we optimize the model generation strategy, dynamically parse database structures, understand natural language instructions, and generate executable SQL statements.
It significantly improves the robustness and accuracy of the model in complex business scenarios, reduces the SQL generation error rate, and enhances the interpretability and adaptability of the generation process.
Smart Images
Figure CN122364249A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of databases, and more particularly to a method and apparatus for training a natural language to SQL model based on reinforcement learning. Background Technology
[0002] Natural Language to SQL (NL2SQL) technology is widely used in enterprise data analysis, business decision support, and intelligent customer service. In practical applications, users typically describe their query requirements in natural language. The system needs to automatically parse the user's intent, retrieve relevant databases, generate executable SQL code, and return the query results. The core value of this technology lies in lowering the barrier to entry for non-technical users, enabling them to efficiently gain data insights without needing to master SQL syntax.
[0003] However, real-world scenarios present numerous challenges: First, enterprise databases are complex, containing multiple tables with foreign key relationships, requiring the model to accurately retrieve relevant table structures and understand the meaning of fields; second, user queries may involve complex logic (such as multi-condition filtering, grouping statistics, and cross-table joins), demanding that the model possess deep thinking capabilities to generate correct SQL statements; finally, dynamic updates to data tables and non-standard field naming in real-world business scenarios further increase the robustness requirements of the model. Summary of the Invention
[0004] This application provides a method and apparatus for training a natural language to SQL model based on reinforcement learning, in order to improve the robustness of large language models.
[0005] In a first aspect, embodiments of this application provide a method for training a natural language to SQL model based on reinforcement learning, the method comprising:
[0006] Receive natural language instructions and retrieve structural information from the target database based on the natural language instructions;
[0007] Based on natural language instructions and structural information, external tools are invoked to generate a response containing the tool invocation history and SQL execution results; the external tools include at least one of search tools, calculation tools, and SQL execution tools.
[0008] Based on the response containing the tool call trajectory and SQL execution results, the SQL execution results are inferred and verified to obtain the verification results. The inference and verification includes comparing the difference between the result table of the SQL generated by the model and the actual table through the SQL validator.
[0009] Based on the validation results, the generation strategy of the large language model was adjusted to generate executable SQL statements.
[0010] In one possible implementation, an external tool is invoked to generate a response containing the tool invocation trace and SQL execution results, based on natural language instructions and structural information, including:
[0011] The search tool is invoked to parse the business terms in the natural language instructions, and the parsing results are obtained.
[0012] Call the calculation tool to generate intermediate variables based on the analysis results;
[0013] Call the SQL execution tool and obtain the SQL execution result based on the intermediate variables.
[0014] In one possible implementation, an SQL execution tool is invoked to obtain the SQL execution result based on intermediate variables, including:
[0015] Call the SQL execution tool to generate the initial SQL statement based on the intermediate variables;
[0016] The SQL execution result is determined by comparing the result table corresponding to the initial SQL statement with the actual table using the EXCEPT and UNION ALL operations.
[0017] In one possible implementation, the generation strategy of the large language model is adjusted based on the verification results, including:
[0018] The first reward value is determined by comparing the SQL result table generated by the model with the actual table.
[0019] Verify the output format of the tool's call trajectory to determine the second reward value;
[0020] The generation strategy of the large language model is adjusted based on the first reward value and the second reward value.
[0021] In one possible implementation, retrieving structural information from a target database based on natural language instructions includes:
[0022] Dynamically parse the metadata of the target database table structure and generate a topology diagram of the table structure;
[0023] Retrieve data tables related to natural language commands based on the topology graph.
[0024] In one possible implementation, after receiving the natural language instruction, the process further includes:
[0025] The natural language understanding module identifies ambiguous terms in natural language instructions.
[0026] Retrieve business rules related to fuzzy terms using knowledge graphs;
[0027] Adjust the semantic parsing logic of natural language instructions based on business rules.
[0028] Secondly, embodiments of this application provide a training apparatus for a natural language to SQL model based on reinforcement learning, the apparatus comprising:
[0029] The input module receives natural language commands and retrieves structural information from the target database based on these commands.
[0030] The processing module is used to invoke external tools based on natural language instructions and structural information, and generate a response containing the tool invocation trace and SQL execution results; the external tools include at least one of search tools, calculation tools, and SQL execution tools;
[0031] The verification module is used to perform inference verification on the SQL execution result based on the response containing the tool call trajectory and the SQL execution result, and obtain the verification result. The inference verification includes comparing the difference between the result table of the SQL generated by the model and the real table through the SQL verifier.
[0032] The optimization module is used to adjust the generation strategy of the large language model based on the verification results in order to generate executable SQL statements.
[0033] Thirdly, embodiments of this application provide a computer device, including: a memory and a processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.
[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.
[0035] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.
[0036] This application provides a reinforcement learning-based natural language to SQL model training method and apparatus. The method includes: receiving natural language instructions and retrieving structural information of a target database based on the natural language instructions; calling an external tool according to the natural language instructions and structural information to generate a response containing the tool call trajectory and SQL execution results; the external tool includes at least one of a search tool, a calculation tool, and an SQL execution tool; performing inference verification on the SQL execution results based on the response containing the tool call trajectory and SQL execution results to obtain verification results, the inference verification including comparing the difference between the result table of the SQL generated by the model and the real table through an SQL validator; and adjusting the generation strategy of the large language model according to the verification results to generate executable SQL statements. An end-to-end natural language to SQL training framework is constructed through reinforcement learning algorithms, combining tool fusion inference and interleaved thinking mechanisms to dynamically optimize the model's decision-making capabilities throughout the entire process of data table retrieval, logical calculation, and SQL generation. Attached Figure Description
[0037] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0038] Figure 1 A flowchart illustrating the reinforcement learning method for natural language to SQL provided in this application. Figure 1 ;
[0039] Figure 2 A flowchart illustrating the reinforcement learning method for natural language to SQL provided in this application. Figure 2 ;
[0040] Figure 3 A schematic diagram of the reinforcement learning device for natural language to SQL provided in this application;
[0041] Figure 4 A schematic diagram of the structure of the computer device provided in this application.
[0042] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0043] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.
[0044] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0045] Natural Language to SQL (NL2SQL) technology is widely used in enterprise data analysis, business decision support, and intelligent customer service. In practical applications, users typically describe their query requirements in natural language. The system needs to automatically parse the user's intent, retrieve relevant databases, generate executable SQL code, and return the query results. The core value of this technology lies in lowering the barrier to entry for non-technical users, enabling them to efficiently gain data insights without needing to master SQL syntax.
[0046] However, real-world scenarios present numerous challenges: First, enterprise databases are complex, containing multiple tables with foreign key relationships, requiring the model to accurately retrieve relevant table structures and understand the meaning of fields; second, user queries may involve complex logic (such as multi-condition filtering, grouping statistics, and cross-table joins), demanding that the model possess deep thinking capabilities to generate correct SQL statements; finally, dynamic updates to data tables and non-standard field naming in real-world business scenarios further increase the robustness requirements of the model.
[0047] To address the aforementioned technical issues, this application provides a reinforcement learning-based natural language to SQL model training method, suitable for enterprise-level data query automation scenarios. For example, business personnel can quickly obtain database results via natural language commands (such as "query the top 10% of customers by sales in 2025"). The system needs to be integrated with databases (such as MySQL and PostgreSQL) and external tools (such as Python sandboxes and search engines), and deployed on a cloud or local server. The network architecture includes: a natural language input interface, receiving user commands and transmitting them to the model; a model processing unit, which calls tools based on a reinforcement learning framework (such as GRPO) and generates SQL trajectories; a database and tool interface, performing SQL verification, Python calculations, and search operations; and a result feedback module, which optimizes the model strategy through rule-based reward functions (such as XML tag matching and result table verification).
[0048] In one embodiment, such as Figure 1 As shown, the training methods for natural language to SQL models based on reinforcement learning include:
[0049] Step 101: Receive natural language instructions and retrieve the structural information of the target database based on the natural language instructions;
[0050] Step 102: Based on the natural language instructions and structural information, invoke external tools to generate a response containing the tool invocation history and SQL execution results; the external tools include at least one of search tools, calculation tools, and SQL execution tools.
[0051] Step 103: Based on the response containing the tool call trajectory and the SQL execution result, perform inference verification on the SQL execution result to obtain the verification result. The inference verification includes comparing the difference between the result table of the SQL generated by the model and the real table through the SQL validator.
[0052] Step 104: Based on the verification results, adjust the generation strategy of the large language model to generate executable SQL statements.
[0053] Natural language commands refer to unstructured query statements entered by users, such as "query the top 10% of customers in terms of sales in 2025".
[0054] Structural information refers to the collection of field names, data types, and relationships between tables in a database. For example, the customer table contains fields customer_id, name, and sales.
[0055] External tools are used to assist large language models in processing natural language instructions for computation or verification modules, including search tools, computation tools, SQL execution tools, etc.
[0056] Tool call traces refer to the complete record of external tool calls made by the large language model during the generation of SQL statements, such as... <search>...<excecute_sql> ...
[0057] Executable SQL statements: Structured query statements that conform to database syntax standards and can be executed directly, such as SELECT * FROM customers WHERE sales > (SELECT PERCENTILE_CONT(0.9) FROM sales).
[0058] First, it receives natural language commands input by the user (e.g., "Query the top 10% of customers by sales") and retrieves relevant data tables based on the database table structure (e.g., customer table, sales table). Then, it calls external tools (e.g., a search tool to parse the calculation logic of "top 10% by sales," and a Python executor to calculate the threshold) to process the command, generating a record containing the tool call history (e.g., ...). <search>...<excecute_py> ...<excecute_sql> ...) and the response to the SQL execution result. It should be noted that the response also includes the generated initial SQL statement. Finally, by comparing the differences between the generated SQL result table and the actual data table, a comparison result is obtained. Reward signals are generated based on the comparison result, such as XML tag matching scores and result table correctness scores. The large language model adjusts its generation strategy based on the reward signals, optimizes the tool's calling logic and SQL generation capabilities, ultimately achieving end-to-end natural language to SQL processing.
[0059] The tool call trace is passed as an intermediate result to subsequent steps, ensuring that the SQL statement generation process is traceable. For example, when a user enters "count employees whose sales are higher than the department average", the model calls the search tool to parse the calculation logic of "department average", calls the Python executor to calculate the average of department groups, and generates an SQL statement containing subqueries.
[0060] The method provided in the above embodiments solves the problem of insufficient generalization ability caused by the reliance on labeled SQL statements in supervised fine-tuning models by combining reinforcement learning and tool invocation. Specifically, before generating SQL, the model autonomously invokes a search tool to parse business terms (such as "top 10% of sales"), a calculation tool performs numerical calculations (such as generating sales thresholds), and an SQL validator provides real-time feedback on the correctness of the result table. Technically, the combination of tool invocation and reinforcement learning guides model strategy optimization through reward signals (such as XML tag matching and result table validation), enabling it to autonomously explore data table structures and query logic, rather than relying on labeled SQL statements. This technique significantly reduces the model illusion rate, improves the interpretability of generated SQL, and solves the problem of unrecognizable equivalent SQL results, achieving an end-to-end NL2SQL system that more closely resembles real-world business scenarios.
[0061] In one embodiment, an external tool is invoked to generate a response containing the tool invocation history and SQL execution results based on natural language instructions and structural information, including:
[0062] The search tool is invoked to parse the business terms in the natural language instructions, and the parsing results are obtained.
[0063] Call the calculation tool to generate intermediate variables based on the analysis results;
[0064] Call the SQL execution tool and obtain the SQL execution result based on the intermediate variables.
[0065] Business terminology refers to domain-specific concepts involved in natural language instructions, such as "top 10% of sales" or "department average."
[0066] Search tools are tools used to parse business terms (such as "top 10% of sales") in natural language instructions and obtain the meaning of related fields. For example, you can use a search engine to query the calculation logic corresponding to "top 10% of sales".
[0067] Computational tools are tools that perform numerical calculations or logical operations. For example, a Python sandbox environment is used to execute `np.percentile(sales, 90)` to generate a sales threshold.
[0068] SQL execution tools are tools that convert intermediate variables into SQL statements and obtain SQL execution results. For example, they can generate a SELECT * FROM customers WHERE sales > {threshold} statement based on field names and calculation logic.
[0069] In the step of calling the tool execution module, the model first calls a search tool to parse the business terms (such as "top 10% of sales") in the natural language instructions, obtaining the meaning of the fields (such as the field name "sales" and the calculation logic). Then, it calls a calculation tool to generate intermediate variables (such as the sales threshold of 1 million) based on the parsing results. Finally, it calls an SQL execution tool to convert the intermediate variables into SQL statements (such as SELECT * FROM customers WHERE sales > 1,000,000). This process ensures the accuracy and interpretability of the generated statements by progressively building the SQL logic through staged tool calls.
[0070] By invoking search, computation, and SQL execution tools in stages, precise matching between tool calls and natural language instructions is achieved. Specifically, the parsing capabilities of the search tool ensure the model understands business terminology, the intermediate variable generation capabilities of the computation tool improve the accuracy of SQL logic, and the transformation capabilities of the SQL execution tool guarantee the executability of the final statement. This technology significantly reduces the SQL error rate caused by misunderstandings of field names or missing logic, and enhances the traceability of the generation process, providing a clearer reasoning chain for subsequent result verification.
[0071] In one embodiment,
[0072] The SQL execution tool is invoked to obtain the SQL execution results based on intermediate variables, including:
[0073] Call the SQL execution tool to generate the initial SQL statement based on the intermediate variables;
[0074] The SQL execution result is determined by comparing the result table corresponding to the initial SQL statement with the actual table using the EXCEPT and UNION ALL operations.
[0075] EXCEPT operation: Compares rows that do not match in two result tables.
[0076] The UNION ALL operation merges all rows from two result tables.
[0077] This step refines the comparison logic of the SQL execution tool. For example, after the initial SQL statement generated by the model is executed, the differences between the result table and the actual table are compared using EXCEPT and UNION ALL operations. If differences exist (such as inconsistent field names), the model adjusts its generation strategy (such as correcting subquery conditions) to ensure that the final generated SQL statement conforms to the actual table structure.
[0078] By employing comparison logic with EXCEPT and UNION ALL operations, the problem of traditional methods being unable to handle equivalent SQL statements is solved. For example, the model can identify semantically equivalent but structurally different SQL statements (such as SELECT * FROM A WHERE id> 10 and SELECT * FROM A WHERE id NOT IN (SELECT id FROM A WHERE id <= 10)), significantly improving the robustness of SQL generation.
[0079] In one embodiment, the generation strategy of the large language model is adjusted based on the verification results, including:
[0080] The first reward value is determined by comparing the SQL result table generated by the model with the actual table.
[0081] Verify the output format of the tool's call trajectory to determine the second reward value;
[0082] The generation strategy of the large language model is adjusted based on the first reward value and the second reward value.
[0083] The first reward value represents the correctness of the result. For example, the correctness of the result can be evaluated using an SQL validator, with 1 point awarded for each correct result.
[0084] The second reward value characterizes the correctness of the format of the output tool call trajectory. Specifically, in one embodiment, when generating the tool call trajectory, the corresponding tools are annotated using XML (eXtensible Markup Language), including thought tags. <think>Search tools <search>Lightweight Python sandbox tool<excecute_py> SQL executor tools<excecute_sql> and results labels <answer>First, using a regular expression parser, match each XML tag in the response, including... <think>(Thinking Tags) <search>(Search tool tags)<excecute_py> (Python execution tool tag)<excecute_sql> (SQL execution tool label) and <answer>(Answer tags), each captured set of tags awards 0.2 points.
[0085] Based on the first and second reward values, the generation strategy of the large language model is adjusted to produce more accurate executable SQL statements.
[0086] In one embodiment, retrieving structural information of a target database based on natural language instructions includes:
[0087] Dynamically parse the metadata of the database table structure and generate a topological diagram of the table structure;
[0088] Retrieve data tables related to natural language commands based on the topology graph.
[0089] Metadata: Information describing the structure of a database table, including field names, data types, index information, etc.
[0090] Topology diagram: A graph structure that represents the relationships between database tables.
[0091] By parsing the metadata of the database table structure (such as field names and index information), a topological graph of the table structure is generated (e.g., the customer table → sales table are related via customer_id), and data tables related to natural language commands are retrieved based on this graph. For example, when a user enters "query employees whose sales are higher than the department's average," the model parses the relationship between the "department table" and the "sales table" and prioritizes retrieving relevant tables.
[0092] By dynamically parsing table structure metadata, the problem of traditional methods being unable to adapt to dynamic changes in table structure (such as changes in field names or the addition of new tables) is solved. For example, when a "Region" field is added to the "Sales" table, the model can automatically retrieve this field and generate SQL statements that include region grouping, significantly improving the model's adaptability to dynamic data environments.
[0093] In one embodiment, such as Figure 2 As shown, after the step of receiving natural language instructions, the process also includes:
[0094] Step 201: Identify ambiguous terms in natural language instructions using the natural language understanding module;
[0095] Step 202: Retrieve business rules related to fuzzy terms using a knowledge graph;
[0096] Step 203: Adjust the semantic parsing logic of natural language instructions based on business rules.
[0097] Ambiguous terms: words in natural language instructions that are ambiguous, such as "recently" which may correspond to "the past week" or "the past month".
[0098] Knowledge graph: A structured knowledge base that stores business rules, such as "sales data is updated daily".
[0099] After receiving natural language instructions, the model adds fuzzy term recognition and business rule retrieval. For example, when a user enters "query the latest sales data", the model uses the natural language understanding module to identify the fuzziness of "latest", then retrieves relevant business rules (such as "sales data update cycle is daily") through the knowledge graph, and finally adjusts the semantic parsing logic (such as parsing "latest" as "the past day").
[0100] By using fuzzy term recognition and business rule retrieval, the problem of traditional methods being unable to handle fuzzy queries is solved. For example, the model can dynamically parse the specific meaning of "recent" and generate SQL statements that conform to business rules, avoiding the generation of erroneous SQL due to ambiguity and significantly improving the reliability of queries in complex business scenarios.
[0101] In one embodiment, in order to fully stimulate the tool fusion reasoning and interleaved thinking capabilities of the base model, and considering that the training effect of mainstream reinforcement learning algorithms (PPO, GRPO, REINFORCE++, etc.) is closely related to policy entropy dynamics, in order to avoid premature entropy collapse, encourage the model to explore fully, and improve the upper limit of reinforcement learning effect, data engineering needs to be specifically designed.
[0102] To provide good policy initialization, a small number of labeled samples are used for a cold start before reinforcement learning. The cold start employs Supervised Fine-Tuning (SFT). These labeled samples contain real inference trajectories, and the corresponding tools are annotated using XML (eXtensible Markup Language), including thought tags. <think>Search tools <search>Lightweight Python sandbox tool<excecute_py> SQL executor tools<excecute_sql> and results labels <answer>The main purpose of the cold start phase is to teach the model the basic answer format, guide the model to adapt to the question pattern, and avoid wasting time on inefficient exploration in the early stages of subsequent reinforcement learning.
[0103] In terms of reinforcement learning sample design, the question-answer pair annotation uses thousands of real, high-quality "natural language instruction-result table" examples. No inference paths are labeled, and no Process Reward Models (PRM) are used. Instead, customized system prompts guide the model to perform tool-integrated reasoning and interleaved thinking. Similar to the cold start phase, this phase also uses XML notation to standardize response patterns and directly supervises rewards for both the results and response patterns.
[0104] In one embodiment, to reduce the accuracy loss caused by quantization during deployment, both training stages use INT8 quantization-aware training.
[0105] The cold start phase uses LORA fine-tuning with a rank of 16. LORA is inserted into the Q, K, and V projection matrices, and the learning rate is 2e-5 with cosine decay. The batch size is 32.
[0106] The reinforcement learning algorithm uses GRPO, with a group size of 32 and a regularization coefficient (Beta) of 0.01. The LORA rank is 16, and the Q, K, and V projection matrices are inserted into LORA. The learning rate is 1e-5 with cosine decay. The batch size is 32.
[0107] To mitigate the training instability caused by routing inconsistency during RL training of Mixture of Experts (MoE) models, even though GRPO already includes KL divergence penalty which can alleviate the problem to some extent, this scheme adds an additional rule-based reward function to encourage the model to select similar experts for similar inputs.
[0108] Specifically, a reinforcement learning training method for natural language to SQL conversion is presented, including:
[0109] Step 1: Prepare the NL2SQL dataset and perform supervised fine-tuning (SFT).
[0110] 1. Collect a small number of NL2SQL labeled samples: Construct hundreds to thousands of samples of "natural language instructions → reasoning trajectory + final answer".
[0111] The reasoning trajectory includes interleaved occurrences <think> 、 <search>,<execute_py> ,<execute_sql> XML tags such as ...
[0112] The final SQL answer is wrapped in <answer>The label contains the correct SQL statement.
[0113] 2. Supervised fine-tuning of the base model: Using the above samples, and employing the standard SFT loss function, the model is initially trained to:
[0114] Basic response format (how to use XML tags).
[0115] Simple tool call sequence (e.g., think first, then call the SQL executor, then give the answer).
[0116] Avoid generating completely random output in the early stages of RL.
[0117] Step 2: Constructing a reinforcement learning training environment
[0118] 1. Define the executable toolkit:
[0119] <search>Simulates a search engine or knowledge base query.
[0120] <execute_py> A lightweight Python sandbox environment that supports numerical computation and data processing.
[0121] <execute_sql> SQL executor, connects to a real or simulated database, and returns a query result table.
[0122] 2. Prepare RL training data: Collect thousands of "natural language instruction - standard answer result table" pairs (excluding reasoning process).
[0123] Each instruction corresponds to an expected result table (TABLE2).
[0124] 3. Set system prompts: At the start of each RL interaction, provide the model with customized system prompts, specifying the requirements:
[0125] Must use <think>,<execute_sql>, <answer>XML tags such as ...
[0126] Interleaving thinking and tool usage is permitted and encouraged.
[0127] The final answer must be an SQL statement, and it must be wrapped in... <answer>Within the label.
[0128] Step 3: Define the reward function (rule-based outcome reward)
[0129] For each complete rollout generated by the strategy model, analyze and calculate the reward:
[0130] 1. Formatting Reward (Correct Formatting)
[0131] Use a regular expression parser to match XML tags in the response: <think> , <search>,<execute_py>,<execute_sql>, <answer>.
[0132] For each successfully captured set of tags (with complete tags and non-empty content), a reward of +0.2 is given, which means the second reward value increases by 0.2.
[0133] The same label appearing multiple times can be scored repeatedly.
[0134] 2. Reward for correct results (matching SQL execution results)
[0135] from <answer>Extract the SQL statement from the tag, execute it in the environment, and get the result table TABLE1.
[0136] Perform a set comparison with the standard answer result table TABLE2;
[0137] If the above query returns empty (the two result tables are exactly the same), a reward of +1.0 is given, that is, the first reward value is increased by 1.0; otherwise, a reward of 0 is given.
[0138] 3. Total reward = First reward value + Second reward value.
[0139] Step 4: Perform reinforcement learning iterative training
[0140] Using a gradient algorithm based on strategies such as PPO, GRPO, or REINFORCE++, repeat the following steps:
[0141] 1. Sampling trajectory:
[0142] Generate a rollout by sampling from the current policy model.
[0143] Each trajectory corresponds to a natural language instruction, and the model autonomously decides when to output it. <think>When to call the tool, how to process the returned results after the tool is called, and when to finally output the results. <answer>.
[0144] 2. Calculate the reward:
[0145] Calculate the total reward for each trajectory according to the reward function in step 3.
[0146] 3. Calculate the advantage estimate:
[0147] Use methods such as GAE to estimate the advantage value of each state-action pair (or each generated token).
[0148] Since rewards are only obtained at the end of the trajectory (result reward) and some intermediate format rewards, credits need to be allocated reasonably.
[0149] 4. Update the strategy model:
[0150] The model parameters are updated by maximizing the clip objective function (the clip objective of PPO).
[0151] It can also update the value network (critic) to reduce the value function estimation error.
[0152] 5. Policy entropy regularization (optional but recommended):
[0153] Adding a policy entropy term to the loss function encourages the model to keep exploring and prevents it from getting trapped in local optima too early (entropy collapse).
[0154] Step 5: Verification and Iteration
[0155] 1. Periodic evaluation: Calculate the accuracy of the SQL execution results generated by the model (the proportion of result tables that are completely matched) on the reserved validation set.
[0156] 2. If the accuracy rate does not meet the target or overfitting occurs:
[0157] Adjust reward weights (e.g., increase format rewards to improve parsability).
[0158] Add more diverse tool call patterns to the cold start data.
[0159] Increase the number of RL training steps or adjust the learning rate.
[0160] 3. Final Output Model: After training convergence, an NL2SQL model with tool fusion reasoning and interleaved thinking capabilities is obtained. When using it, only system prompts and natural language commands are required, and the model will automatically generate a model containing... <answer>SQL statements for tags.
[0161] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0162] Based on the same inventive concept, this application also provides a reinforcement learning-based natural language to SQL model training device. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more reinforcement learning-based natural language to SQL model training device embodiments provided below can be found in the limitations of the reinforcement learning-based natural language to SQL model training method above, and will not be repeated here.
[0163] In one embodiment, such as Figure 3 As shown, the reinforcement learning-based natural language to SQL model training device includes an input module 301, a processing module 302, a verification module 303, and an optimization module 304, wherein:
[0164] The input module 301 receives natural language instructions and retrieves structural information of the target database based on the natural language instructions;
[0165] The processing module 302 is used to invoke external tools based on natural language instructions and structural information, and generate a response containing the tool invocation trace and SQL execution results; the external tools include at least one of search tools, calculation tools and SQL execution tools;
[0166] The verification module 303 is used to perform inference verification on the SQL execution result based on the response containing the tool call trajectory and the SQL execution result, and obtain the verification result. The inference verification includes comparing the difference between the result table of the SQL generated by the model and the real table through the SQL verifier.
[0167] The optimization module 304 is used to adjust the generation strategy of the large language model based on the verification results in order to generate executable SQL statements.
[0168] In one possible implementation, the processing module 302 is specifically used for:
[0169] The search tool is invoked to parse the business terms in the natural language instructions, and the parsing results are obtained.
[0170] Call the calculation tool to generate intermediate variables based on the analysis results;
[0171] Call the SQL execution tool and obtain the SQL execution result based on the intermediate variables.
[0172] In one possible implementation, the processing module 302 is specifically used for:
[0173] Call the SQL execution tool to generate the initial SQL statement based on the intermediate variables;
[0174] The SQL execution result is determined by comparing the result table corresponding to the initial SQL statement with the actual table using the EXCEPT and UNION ALL operations.
[0175] In one possible implementation, optimization module 304 is specifically used for:
[0176] The first reward value is determined by comparing the SQL result table generated by the model with the actual table.
[0177] Verify the output format of the tool's call trajectory to determine the second reward value;
[0178] The generation strategy of the large language model is adjusted based on the first reward value and the second reward value.
[0179] In one possible implementation, the input module 301 is specifically used for:
[0180] Dynamically parse the metadata of the database table structure and generate a topological diagram of the table structure;
[0181] Retrieve data tables related to natural language commands based on the topology graph.
[0182] In one possible implementation, the input module 301 is specifically used for:
[0183] The natural language understanding module identifies ambiguous terms in natural language instructions.
[0184] Retrieve business rules related to fuzzy terms using knowledge graphs;
[0185] Adjust the semantic parsing logic of natural language instructions based on business rules.
[0186] The modules in the aforementioned reinforcement learning device for natural language to SQL conversion can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0187] Figure 4 A schematic diagram of the structure of the computer device provided in this application. Figure 4 As shown, the computer device 40 provided in this embodiment includes at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, memory 402, and communication component 403 are connected via a bus 404.
[0188] In a specific implementation, at least one processor 401 executes computer execution instructions stored in memory 402, causing at least one processor 401 to perform the above-described method.
[0189] The specific implementation process of processor 401 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.
[0190] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.
[0191] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.
[0192] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0193] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0194] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.
[0195] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.
[0196] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.
[0197] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.
[0198] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0199] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.
[0200] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0201] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0202] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.< / answer> < / answer> < / think> < / answer> < / answer> < / search> < / think> < / answer> < / answer> < / think> < / search> < / answer> < / search> < / think> < / answer> < / search> < / think> < / answer> < / search> < / think> < / answer> < / search> < / think> < / search> < / search>
Claims
1. A method for training a natural language to SQL model based on reinforcement learning, characterized in that, The method includes: Receive natural language instructions and retrieve structural information of the target database based on the natural language instructions; Based on the natural language instructions and the structural information, external tools are invoked to generate a response containing the tool invocation history and SQL execution results; the external tools include at least one of search tools, calculation tools, and SQL execution tools. Based on the response containing the tool call trajectory and SQL execution result, the SQL execution result is inferred and verified to obtain the verification result. The inference and verification includes comparing the difference between the result table of the SQL generated by the model and the real table through the SQL validator. Based on the verification results, the generation strategy of the large language model is adjusted to generate executable SQL statements.
2. The method according to claim 1, characterized in that, The process of calling external tools and generating a response containing the tool call history and SQL execution results includes: The search tool is invoked to parse the business terms in the natural language instructions, and the parsing results are obtained. The computational tool is invoked to generate intermediate variables based on the analysis results; The SQL execution tool is invoked, and the SQL execution result is obtained based on the intermediate variables.
3. The method according to claim 2, characterized in that, The process of calling the SQL execution tool and obtaining the SQL execution result based on the intermediate variables includes: The SQL execution tool is invoked to generate an initial SQL statement based on the intermediate variables; The SQL execution result is determined by comparing the result table corresponding to the initial SQL statement with the actual table using the EXCEPT and UNION ALL operations.
4. The method according to claim 1, characterized in that, The step of adjusting the generation strategy of the large language model based on the verification results includes: The first reward value is determined by comparing the SQL result table generated by the model with the actual table. Verify the output format of the tool's call trajectory to determine the second reward value; The generation strategy of the large language model is adjusted based on the first reward value and the second reward value.
5. The method according to claim 1, characterized in that, The structural information of the target database retrieved based on the natural language instructions includes: Dynamically parse the metadata of the target database table structure and generate a topology diagram of the table structure; Retrieve data tables related to the natural language instructions based on the topology graph.
6. The method according to claim 1, characterized in that, After receiving the natural language instruction, the process also includes: The natural language understanding module identifies ambiguous terms in the natural language instructions. Retrieve business rules related to the fuzzy terms using a knowledge graph; The semantic parsing logic of natural language instructions is adjusted based on the aforementioned business rules.
7. A training device for a natural language to SQL model based on reinforcement learning, characterized in that, The device includes: The input module receives natural language instructions and retrieves structural information from the target database based on the natural language instructions. The processing module is configured to invoke external tools based on the natural language instructions and the structural information, and generate a response containing the tool invocation history and SQL execution results; the external tools include at least one of search tools, calculation tools, and SQL execution tools; The verification module is used to perform inference verification on the SQL execution result based on the response containing the tool call trajectory and the SQL execution result, and obtain the verification result. The inference verification includes comparing the difference between the result table of the SQL generated by the model and the real table through the SQL verifier. The optimization module is used to adjust the generation strategy of the large language model based on the verification results in order to generate executable SQL statements.
8. A computer device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 6.
10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.