NL2SQL reinforcement learning method based on multi-round SQL perception reward
By employing a reinforcement learning method based on multi-turn SQL-aware rewards, this approach addresses the issues of sparse reward signals and insufficient support for multi-turn dialogue in existing NL2SQL methods. It achieves more efficient training and more accurate SQL generation, thereby enhancing the performance of the agent in complex scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NINGBO TELIAN INFORMATION TECH CO LTD
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-26
AI Technical Summary
Existing reinforcement learning-based NL2SQL methods suffer from sparse reward signals, resulting in low training efficiency, and lack of deep modeling of multi-turn dialogues, leading to poor generalization performance and semantic drift issues in complex scenarios.
We employ a reinforcement learning approach based on multi-round SQL-aware rewards. By acquiring multi-round dialogue trajectory data from ended sessions, we calculate dense reward values and total reward values. We then use GRPO or PPO optimization strategies to update model parameters and introduce round decay weights to provide fine-grained feedback signals.
It improves the training efficiency and SQL generation accuracy of the model in multi-round interaction scenarios, and enhances its robustness and generalization ability in agent applications.
Smart Images
Figure CN122285690A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and natural language processing, specifically to the field of large-scale natural language to SQL model technology based on reinforcement learning, and specifically provides an NL2SQL reinforcement learning method based on multi-round SQL-aware rewards. Background Technology
[0002] With the explosive development of artificial intelligence technology, especially the rapid iteration and application of Large Language Models (LLM) and intelligent agents, Natural Language to Structured Query Language (NL2SQL) technology is gradually becoming the core technology engine for building data-driven intelligent applications. NL2SQL significantly lowers the barrier for non-technical users to access structured data by accurately parsing users' natural language descriptions into directly executable SQL statement code. In the current context of the booming application of intelligent agents, users often clarify their needs through continuous multi-turn interactions, which poses new challenges to the contextual semantic modeling, cross-turn referential understanding, and long-range state memory capabilities of large models.
[0003] However, most existing reinforcement learning-based NL2SQL methods employ a binary reward mechanism based on SQL execution results (i.e., 1 for correct execution and 0 for incorrect execution). This reward is too sparse, resulting in insufficient signal feedback during model training. Especially when the SQL generation logic is correct, the model struggles to extract fine-grained and effective feedback information from the rewards, leading to low training efficiency and poor generalization performance across database patterns or complex query scenarios. Furthermore, existing methods are mostly focused on single-turn queries, lacking deep modeling of multi-turn dialogue contexts. This makes the model highly susceptible to semantic drift, ambiguous referencing, or intent tracking failures when handling continuous interactions, significantly reducing the accuracy of SQL generation in subsequent rounds and hindering the ability of intelligent agents to make continuous decisions in complex scenarios.
[0004] Accordingly, there is a need in this field for a novel NL2SQL reinforcement learning scheme based on multi-round SQL-aware rewards to address the above problems. Summary of the Invention
[0005] To overcome the above-mentioned shortcomings, this invention is proposed to provide a multi-turn SQL-aware reward-based NL2SQL reinforcement learning method that solves, or at least partially solves, the technical problems of sparse reward signals and insufficient support for multi-turn dialogue in existing reinforcement learning-based NL2SQL methods.
[0006] In a first aspect, the present invention provides an NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, the method comprising: Obtain sampled trajectory data of multi-turn dialogues that have ended; Based on the sampled trajectory data, determine the multi-round dense reward value and total reward value of the ended session; Based on the multi-round dense reward values and total reward values of the ended sessions, the optimized NL2SQL model is selectively output.
[0007] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-turn SQL-aware rewards, the step of obtaining the sampling trajectory data of multi-turn dialogues that have ended includes: Retrieve historical data for the current round of the conversation and dialogue data for the current round of the conversation; Based on the historical data of the current round of the conversation and the dialogue data of the current round of the conversation, a large model of the current round is generated and returned. Based on the large model return results for the current round, obtain the execution results; Obtain the result of whether the session has ended, and based on the result, selectively generate sample trajectory data of multi-turn dialogues in ended sessions.
[0008] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, the acquisition of historical data of the current round of the session and dialogue data of the current round of the session includes: Retrieve the dialogue data for the current round of the session; Obtain the historical context information of the current round of the session. The historical context information includes at least the user query, all SQL statements generated by the large model before the current round, the execution result of each SQL statement, and the database environment feedback information corresponding to each execution result. The historical context information of the current round of the session is used as the historical data of the current round of the session.
[0009] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, obtaining the execution result based on the large model return result of the current round includes: Extract the SQL statements from the large model return results of the current round; Execute SQL statements in a database environment and obtain the results of the SQL statement execution.
[0010] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, the determination result of whether the session has ended includes: If the number of rounds in the current round of the session reaches the preset maximum number of rounds, or if the duration of the current round of the session reaches the preset duration threshold, then the session is considered to have ended. Otherwise, it is determined that the session has not ended.
[0011] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-turn SQL-aware rewards, the step of selectively generating sampling trajectory data of multi-turn dialogues that have ended, based on the judgment result, includes: If the determination is yes, then the session is regarded as an ended session, and all generated data in each round of the ended session is extracted, and the extracted data is used as the sampling trajectory data of the multi-round dialogue of the ended session; If the determination is negative, the number of rounds will be adjusted, and the historical data of the session will be updated based on all generated data of the current round. Then, the "get historical data of the current round of the session" and subsequent steps will be re-executed.
[0012] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, determining the multi-round dense reward value and the total reward value of the ended session based on the sampled trajectory data includes: Based on the sampled trajectory data, the dense reward value and round decay weight corresponding to each round are determined; The total reward value of the ended session is determined based on the dense reward value corresponding to each round and the round decay weight.
[0013] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, the selective output of the optimized NL2SQL model based on the multi-round dense reward values and the total reward value of the ended session includes: Based on the multi-round dense reward value, total reward value, and preset GRPO optimization strategy of the ended session, the loss function value is determined; Based on the loss function value, the optimized NL2SQL model is selectively output.
[0014] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, the selective output of the optimized NL2SQL model based on the loss function value includes: Based on the loss function value, determine whether the loss training has reached the preset training convergence condition: If the determination is yes, then end the training and output the large model with the current model parameters as the optimized NL2SQL model; If the result is negative, update the model parameters and re-execute the training.
[0015] In one technical solution of the above-mentioned NL2SQL reinforcement learning method based on multi-round SQL-aware rewards, the step of determining the dense reward value and round decay weight corresponding to each round based on the sampled trajectory data includes: The density reward value for each round is obtained using the following formula: In the formula, Representing the The dense reward value of each round, Representing a real example, Representative of false positives Representative of false negatives; The round decay weight corresponding to each round is obtained by the following formula: In the formula, Representing the Round decay weight values for each round, This represents the total number of dialogue rounds for the current sample across multiple sampling rounds. This represents the current round.
[0016] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects: By acquiring sampling trajectory data of multi-turn dialogues in ended sessions, multi-turn dialogues are sampled. Then, by calculating the dense reward values and total reward values of the ended sessions, the model obtains finer-grained feedback signals in multi-turn interaction scenarios, thereby improving training efficiency, SQL generation accuracy, and robustness in agent applications. Based on the dense reward values and total reward values of the ended sessions, an optimized NL2SQL model is selectively output, realizing the construction of a reinforcement learning reward function and training method adapted to multi-turn dialogues, namely, optimization based on group relative policies (…). Using GRPO (or Proximal Policy Optimization) or PPO as the optimization framework, this invention models the multi-turn dialogue NL2SQL task as a multi-turn sequential decision-making process. By introducing turn decay weights, it assigns slightly greater incentives to sessions that are efficiently resolved in early-turn rounds. This invention effectively solves the problem of maintaining contextual semantics across rounds and provides fine-grained feedback signals to the model. Compared with existing technologies, this method can effectively improve training convergence efficiency and SQL generation accuracy in complex multi-turn scenarios, giving the model stronger robustness and generalization ability in continuous agent interaction applications. Attached Figure Description
[0017] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein: Figure 1 This is a flowchart illustrating the main steps of an NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the process for acquiring sampling trajectory data using an NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the model optimization process of an NL2SQL reinforcement learning method based on multi-round SQL-aware reward according to an embodiment of the present invention; Detailed Implementation
[0018] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.
[0019] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0020] See appendix Figure 1 , Figure 1 This is a schematic diagram illustrating the main steps of an NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to an embodiment of the present invention. Figures 1 to 3 As shown, the NL2SQL reinforcement learning method based on multi-round SQL-aware reward in this embodiment of the invention mainly includes the following steps S101-S103.
[0021] Step S101: Obtain the sampling trajectory data of multi-turn dialogues that have ended; Specifically, obtaining the sampling trajectory data of multi-turn dialogues that have ended includes: Retrieve historical data for the current round of the conversation and dialogue data for the current round of the conversation; Based on the historical data of the current round of the conversation and the dialogue data of the current round of the conversation, a large model of the current round is generated and returned. Based on the large model return results for the current round, obtain the execution results; Obtain the result of whether the session has ended, and based on the result, selectively generate sample trajectory data of multi-turn dialogues in ended sessions.
[0022] Specifically, obtaining the historical data of the current round of the session and the dialogue data of the current round of the session includes: Retrieve the dialogue data for the current round of the session; Obtain the historical context information of the current round of the session. The historical context information includes at least the user query, all SQL statements generated by the large model before the current round, the execution result of each SQL statement, and the database environment feedback information corresponding to each execution result. The historical context information of the current round of the session is used as the historical data of the current round of the session.
[0023] Specifically, the process of obtaining the execution result based on the large model return result for the current round includes: Extract the SQL statements from the large model return results of the current round; Execute SQL statements in a database environment and obtain the results of the SQL statement execution.
[0024] Specifically, the determination result of whether the session has ended includes: If the number of rounds in the current round of the session reaches the preset maximum number of rounds, or if the duration of the current round of the session reaches the preset duration threshold, then the session is considered to have ended. Otherwise, it is determined that the session has not ended.
[0025] Specifically, the preset maximum number of rounds and the preset duration threshold can be set by those skilled in the art according to the actual situation, and will not be elaborated here.
[0026] Specifically, the selective generation of sampled trajectory data for multi-turn dialogues that have ended, based on the judgment result, includes: If the determination is yes, then the session is regarded as an ended session, and all generated data in each round of the ended session is extracted, and the extracted data is used as the sampling trajectory data of the multi-round dialogue of the ended session; If the determination is negative, the number of rounds will be adjusted, and the historical data of the session will be updated based on all generated data of the current round. Then, the "get historical data of the current round of the session" and subsequent steps will be re-executed.
[0027] Specifically, the number of adjustment rounds is: number of rounds + 1.
[0028] Specifically, updating the session's historical data based on all generated data from the current round includes: All generated data in the current round is stored in the session's historical data, thus updating the session's historical data.
[0029] Step S102: Based on the sampled trajectory data, determine the multi-round dense reward value and total reward value of the ended session; Specifically, determining the multi-round dense reward value and total reward value of the ended session based on the sampled trajectory data includes: Based on the sampled trajectory data, the dense reward value and round decay weight corresponding to each round are determined; The total reward value of the ended session is determined based on the dense reward value corresponding to each round and the round decay weight.
[0030] Specifically, determining the dense reward value and round decay weight for each round based on the sampled trajectory data includes: The density reward value for each round is obtained using the following formula: In the formula, Representing the The dense reward value of each round, Representing a real example, Representative of false positives Representative of false negatives; The round decay weight corresponding to each round is obtained by the following formula: In the formula, Representing the Round decay weight values for each round, This represents the total number of dialogue rounds for the current sample across multiple sampling rounds. This represents the current round.
[0031] Specifically, in some embodiments, the query result of the predicted SQL statement is assumed to be... The actual SQL query result corresponding to the sample is ,So: TP, True Positive, a true example: It exists in and with The number of completely identical records (columns) in the data, that is, rows that are completely matched row by row and field by field, are both correctly generated and truly belong to the real results; FP, False Positive: It exists in, but The number of records (columns) that do not exist in the model, i.e., the "illusion" rows or error rows (false alarms) generated by the model, which do not belong to the true results; FN, False Negative: It exists in, but The number of records (columns) that do not exist in the model, i.e. rows that the model missed (missed detections), these rows should have appeared but were not covered by the generated SQL.
[0032] Specifically, in some embodiments, dense reward values also include execution accuracy, SQL semantic similarity or edit distance, syntax validity, execution efficiency (query latency), etc., assuming there are K types of reward values. The weight of each reward is preset as follows: The dense reward value is .
[0033] Specifically, based on the dense reward value corresponding to each round and the round decay weight, the total reward value of the ended session is determined as follows: The total reward value is obtained using the following formula: In the formula, This represents the total number of dialogue rounds for the current sample during the Rollout process.<User, Assistant> The total number of dialogue pairs can be preset to a fixed maximum number of rounds, which can range from 5 to 10 rounds, or determined by those skilled in the art based on the actual dialogue dynamics. Index representing the turn of the current conversation ( ), This indicates the first round of user initial queries. Representing the The dense reward obtained by the SQL generated by the round model This represents the efficiency decay weight for each round, ensuring that earlier rounds (e.g., t=1) have higher weights, while the weights gradually decrease in later rounds. This guides the model to prioritize solving problems in fewer rounds of interaction, and this weight satisfies the normalization condition. And it decreases with each round.
[0034] Step S103: Based on the multi-round dense reward value and total reward value of the ended session, selectively output the optimized NL2SQL model.
[0035] Specifically, the selective output of the optimized NL2SQL model based on the multi-round dense reward values and total reward values of the ended sessions includes: Based on the multi-round dense reward value, total reward value, and preset GRPO optimization strategy of the ended session, the loss function value is determined; Based on the loss function value, the optimized NL2SQL model is selectively output, either by updating the model parameters or by outputting a larger model with the current model parameters, so as to achieve the optimal value of the model parameters in the output larger model.
[0036] Specifically, the preset GRPO optimization strategy can adopt the existing group-relative strategy optimization GRPO method in the prior art, and its loss function formula can be: In the formula, Indicates distribution from prompts The input obtained by sampling in the middle, This indicates that the current policy model is in response to a given input. Time-generated output The probability, This represents the probability that the old policy model will produce the same output; This indicates the number of samples obtained under the same input. 1 candidate output, Indicates the number of samples within a group; This represents the policy probability ratio, used to measure the change in probability of the same output between the old and new policies; The within-group standardized advantage function is usually obtained by normalizing the reward value and is used to measure the relative quality of the current output relative to the outputs of the same group. This represents the truncation function, used to limit the magnitude of policy updates, where... The cutoff range parameter; The KL divergence between the current policy and the reference policy is used to constrain the stability of policy updates. These are the weighting coefficients for the KL regularization term; The selection of the preset GRPO optimization strategy here is only an example. In actual applications, those skilled in the art can set it themselves, and it will not be elaborated here.
[0037] Specifically, updating model parameters can be performed by the Adam optimizer, and the parameter update formula can be: In the formula, The gradient represents the parameter gradient, specifically through the loss function. The backpropagation was calculated to obtain... This represents the updated parameters. This represents the parameters before the update. This represents the decaying learning rate, which decreases by 5% per round, with an initial decaying learning rate set to... , This represents the gradient clipping function, which clips the gradient vector. Each element is restricted to the interval [-c, c], that is, if the gradient in a certain dimension... Then it is truncated to ; like Then it is truncated to ,in, This represents the gradient clipping threshold, set here to [value]. ; The optimizer selection for updating model parameters here is merely an illustrative example. In practical applications, those skilled in the art can set it themselves, and it will not be elaborated further here.
[0038] Specifically, the selective output of the optimized NL2SQL model based on the loss function value includes: Based on the loss function value, determine whether the loss training has reached the preset training convergence condition: If the determination is yes, then end the training and output the large model with the current model parameters as the optimized NL2SQL model; If the determination is negative, the model parameters are updated and training is re-executed to optimize the large model.
[0039] Specifically, the preset training convergence condition can be the experimental parameter Epoch=5, or the loss difference can be lower than a preset threshold. The preset training convergence condition can be set by those skilled in the art according to the actual situation, and will not be elaborated here.
[0040] Based on steps S101-S103 above, sampling trajectory data generated during multi-turn dialogue interaction is acquired to sample the multi-turn dialogue process. The multi-turn dense reward value for the corresponding session is calculated based on the sampled trajectory data, and the rewards for each round are weighted according to the round decay weight to obtain the total reward value, enabling the model to obtain more granular feedback signals in multi-turn interaction scenarios. Further, based on the multi-turn dense reward value and the total reward value, the model parameters are updated using the Group Relative Policy Optimization (GRPO) algorithm or the Proximal Policy Optimization (PPO) algorithm, thereby realizing a reinforcement learning training method suitable for multi-turn dialogue scenarios. In this method, the multi-turn dialogue NL2SQL task is modeled as a multi-turn sequential decision process, and by introducing round decay weights, sessions that complete the query task in earlier rounds receive higher incentives, effectively solving the problem of cross-round context semantic maintenance and providing the model with more granular reward feedback. Compared to existing technologies, this method improves model training convergence efficiency and SQL generation accuracy in complex multi-turn scenarios, and enhances the robustness and generalization ability of the model in continuous agent interaction applications. It addresses the technical problems of sparse reward signals and insufficient support for multi-turn dialogues in existing reinforcement learning-based NL2SQL methods.
[0041] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.
[0042] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.
[0043] Furthermore, the present invention also provides a control device. In one embodiment of the control device according to the present invention, the control device includes a processor and a storage device. The storage device can be configured to store a program for executing the NL2SQL reinforcement learning method based on multi-round SQL-aware reward of the above-described method embodiments. The processor can be configured to execute the program in the storage device, which includes, but is not limited to, a program for executing the NL2SQL reinforcement learning method based on multi-round SQL-aware reward of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. This control device can be a control device formed by various electronic devices.
[0044] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program that executes the NL2SQL reinforcement learning method based on multi-round SQL-aware rewards described in the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described NL2SQL reinforcement learning method based on multi-round SQL-aware rewards. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0045] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, a part of its hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.
[0046] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.
[0047] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A multi-round SQL-aware reward-based NL2SQL reinforcement learning method, characterized in that, The method includes: Obtain sampled trajectory data of multi-turn dialogues that have ended; Based on the sampled trajectory data, determine the multi-round dense reward value and total reward value of the ended session; Based on the multi-round dense reward values and total reward values of the ended sessions, the optimized NL2SQL model is selectively output.
2. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 1, characterized in that, The acquisition of sampling trajectory data for multi-turn dialogues that have ended includes: Retrieve historical data for the current round of the conversation and dialogue data for the current round of the conversation; Based on the historical data of the current round of the conversation and the dialogue data of the current round of the conversation, a large model of the current round is generated and returned. Based on the large model return results for the current round, obtain the execution results; Obtain the result of whether the session has ended, and based on the result, selectively generate sample trajectory data of multi-turn dialogues in ended sessions.
3. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 2, characterized in that, The acquisition of historical data for the current round of the session and dialogue data for the current round of the session includes: Retrieve the dialogue data for the current round of the session; Obtain the historical context information of the current round of the session. The historical context information includes at least the user query, all SQL statements generated by the large model before the current round, the execution result of each SQL statement, and the database environment feedback information corresponding to each execution result. The historical context information of the current round of the session is used as the historical data of the current round of the session.
4. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 3, characterized in that, The large model return result based on the current round, and the execution result acquisition includes: Extract the SQL statements from the large model return results of the current round; Execute SQL statements in a database environment and obtain the results of the SQL statement execution.
5. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 4, characterized in that, The determination result of whether the session has ended includes: If the number of rounds in the current round of the session reaches the preset maximum number of rounds, or if the duration of the current round of the session reaches the preset duration threshold, then the session is considered to have ended. Otherwise, it is determined that the session has not ended.
6. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 5, characterized in that, The selective generation of sampling trajectory data for multi-turn dialogues that have ended, based on the judgment result, includes: If the determination is yes, then the session is regarded as an ended session, and all generated data in each round of the ended session is extracted, and the extracted data is used as the sampling trajectory data of the multi-round dialogue of the ended session; If the determination is negative, the number of rounds will be adjusted, and the historical data of the session will be updated based on all generated data of the current round. Then, the "get historical data of the current round of the session" and subsequent steps will be re-executed.
7. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 1, characterized in that, The determination of the multi-round dense reward value and total reward value of the ended session based on the sampled trajectory data includes: Based on the sampled trajectory data, the dense reward value and round decay weight corresponding to each round are determined; The total reward value of the ended session is determined based on the dense reward value corresponding to each round and the round decay weight.
8. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 1, characterized in that, The selective output of the optimized NL2SQL model based on the multi-round dense reward values and total reward values of the ended sessions includes: Based on the multi-round dense reward value, total reward value, and preset GRPO optimization strategy of the ended session, the loss function value is determined; Based on the loss function value, the optimized NL2SQL model is selectively output.
9. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 8, characterized in that, The selective output of the optimized NL2SQL model based on the loss function value includes: Based on the loss function value, determine whether the loss training has reached the preset training convergence condition: If the determination is yes, then end the training and output the large model with the current model parameters as the optimized NL2SQL model; If the result is negative, update the model parameters and re-execute the training.
10. The NL2SQL reinforcement learning method based on multi-round SQL-aware rewards according to claim 7, characterized in that, The determination of the dense reward value and round decay weight for each round based on the sampled trajectory data includes: The density reward value for each round is obtained using the following formula: In the formula, Representing the The dense reward value of each round, Representing a real example, Representative of false positives Representative of false negatives; The round decay weight corresponding to each round is obtained by the following formula: In the formula, Representing the Round decay weight values for each round, This represents the total number of dialogue rounds for the current sample across multiple sampling rounds. This represents the current round.