NL2SQL model query enhancement method, device and medium based on reinforcement learning

Abstract

This application discloses a reinforcement learning-based NL2SQL model query enhancement method, device, and medium, relating to the field of data query. The method includes: obtaining multiple existing natural query languages, converting the natural query languages into SQL query statements using the NL2SQL model, and executing the query statements. Based on the execution results, the method determines the context information corresponding to the structural information, as well as the performance of the context information in pre-set multi-dimensional indicators, to generate training samples; based on a group relative strategy optimization algorithm, the method trains the reinforcement learning model using the training samples according to a pre-set reward function; the reward function is set based on the performance of the multi-dimensional indicators of the context information corresponding to the natural query language and the enhanced query language. The reinforcement learning model and the NL2SQL model form a query enhancement framework, achieving efficient and accurate conversion of natural language queries into SQL statements, breaking through the bottleneck of relying on massive amounts of supervised data and poor cross-scenario adaptability.

Description

Technical Field

[0001] The present application relates to the field of data query, and specifically to a method, device, and medium for query enhancement of an NL2SQL model based on reinforcement learning. Background Art

[0002] Currently, information retrieval technology based on the Large Language Model (LLM) has been applied in the field of data retrieval. In terms of structured data retrieval, it is reflected in the recognition of the intention of user natural language queries through the Natural Language to Structured Query Language (NL2SQL) model, and the generation of corresponding Structured Query Language (SQL) statements. The large language model is then associated with the relational database to collaboratively retrieve relevant data information.

[0003] However, in traditional solutions, the retrieval quality of the NL2SQL model is challenged by many factors, including: ambiguity of natural language queries, complex database schemas, semantic alignment issues, insufficient training data, etc.

[0004] To address these issues, retrieval-augmented generation (RAG) can improve the retrieval quality of the NL2SQL model. Query augmentation leverages a large language model to expand or rewrite user queries to better capture relevant information. By bridging the semantic gap between user information needs and document collections, query augmentation helps retrieve more relevant content.

[0005] However, in traditional solutions, query enhancement methods based on large language models usually adopt supervised learning or distillation technology, which requires a lot of computing resources to generate supervision data. In addition, the quality of model training depends largely on the quality and coverage of supervision data, resulting in uncontrollable model quality. Summary of the Invention

[0006] To solve the above problems, this application proposes an NL2SQL model query enhancement method based on reinforcement learning, including:

[0007] Determining a reinforcement learning model, wherein the reinforcement learning model is used to perform a query enhancement task, wherein the query enhancement task is used to output corresponding enhanced query language based on a natural query language input by a user;

[0008] Obtain multiple existing natural query languages, convert the natural query languages into SQL query statements using the NL2SQL model, and execute the SQL query statements. Based on the execution results, determine the context information corresponding to the structural information and the performance of the context information in pre-set multi-dimensional indicators to generate training samples.

[0009] Based on a group relative strategy optimization algorithm, the reinforcement learning model is trained through the training samples according to a preset reward function; the reward function is set based on the performance of multi-dimensional indicators of context information corresponding to the natural query language and the enhanced query language respectively.

[0010] In one example, before obtaining the training samples generated according to the natural query language, the method further includes:

[0011] Define context information based on the database structure information and integrate it to obtain the corresponding context set;

[0012] Define the query enhancement task of the reinforcement learning model, set the natural query language as the state and the enhanced query language as the action;

[0013] Establishing an end-to-end training framework; the end-to-end training framework includes a group relative policy optimization algorithm and a reward function, the group relative policy optimization algorithm generates reasoning steps in the inference phase and generates an enhanced query language in the output phase; the end-to-end training framework performs iterative training based on the reward function;

[0014] Establishing a query scenario; there are multiple query scenarios, and the structural information of the database in different query scenarios is different, which is used to test the universality of the reinforcement learning model.

[0015] In one example, the reward function includes a general part and a separate part, the general part includes multiple first indicators for evaluating the performance of the natural query language and the enhanced query language in terms of the first indicators; the separate part includes multiple second indicators for evaluating the performance of the enhanced query language in terms of the second indicators;

[0016] The first indicators include accuracy, recall rate, and F1 index, and the second indicators include query time and whether an error is reported;

[0017] The second indicator is provided with a corresponding weight factor.

[0018] In one example, based on a group relative policy optimization algorithm and according to a preset reward function, the reinforcement learning model is trained using the training samples, specifically including:

[0019] Determine the corresponding reward function based on the group relative strategy optimization algorithm;

[0020] Based on the general portion of the reward function, at least part of the first indicator is selected as a current indicator, and based on the current indicator, the reinforcement learning model is trained using the training samples until a training expectation is met; the training expectation includes a predetermined number of training rounds and / or a performance of an enhanced query language output by the reinforcement learning model meets a predetermined standard;

[0021] Continue to select at least some of the remaining first indicators, add them to the current indicators, and train the reinforcement learning model based on the added current indicators until all the first indicators are added to the current indicators and the training expectations are met;

[0022] The second indicator is added to the current indicator, and based on the training process of the first indicator, a weight factor corresponding to the second indicator is set, and the reinforcement learning model is trained according to the current indicator.

[0023] In one example, based on the training process of the first indicator, setting the weight factor corresponding to the second indicator specifically includes:

[0024] For each round of training of the first indicator, weighted summation is performed based on the corresponding single-round training efficiency and the corresponding single-round performance to obtain the corresponding overall training efficiency and overall performance; wherein, when the current indicator changes, the next round of training begins, and the later the training process, the higher the weight;

[0025] A weight factor corresponding to the second indicator is determined based on the overall training efficiency and the overall performance; wherein, the higher the overall training efficiency and the better the overall performance, the higher the weight factor corresponding to the second indicator.

[0026] In one example, the method further includes:

[0027] During the training process of the reinforcement learning model, if it is determined that the length of the structural enhancement information in the enhanced query language is longer than a preset length, and the performance of a specified dimension indicator in the corresponding multi-dimensional indicators according to the execution result obtained by the enhanced query language is lower than a preset standard, then determining the specified keyword corresponding to the structural enhancement information in the natural query language according to the structural enhancement information in the enhanced query language;

[0028] Using a large language model, the designated keywords are graded to determine the core keywords and flexible keywords contained therein;

[0029] The structural enhancement information corresponding to the core keywords is retained, and the structural enhancement information corresponding to at least part of the elastic keywords is restored to the corresponding elastic keywords, so as to obtain an updated enhanced query language.

[0030] In one example, restoring the structural enhancement information corresponding to at least some elastic keywords to the corresponding elastic keywords to obtain an updated enhanced query language specifically includes:

[0031] Among all the elastic keywords, a corresponding number of elastic keywords are selected as keyword combinations according to the number of selections corresponding to the current training batch, and the structural enhancement information corresponding to the keyword combinations is restored to the corresponding elastic keywords to obtain an updated enhanced query language and obtain its corresponding performance;

[0032] For all keyword combinations in each training batch, the updated enhanced query language corresponding to several keyword combinations with the best performance of the specified dimension indicators is used to calculate the corresponding reward function and used to train the reinforcement learning model; the training progress of the reinforcement learning model is positively correlated with the number of selected keyword combinations.

[0033] In one example, establishing a query scenario specifically includes:

[0034] During the training of the reinforcement learning model, determining the most common context information corresponding to each execution result in the current query scenario;

[0035] Based on the most common pieces of context information, alternative structural information is generated for substitution to generate a new query scenario for training the reinforcement learning model.

[0036] On the other hand, this application also proposes a NL2SQL model query enhancement device based on reinforcement learning, including:

[0037] at least one processor; and,

[0038] a memory communicatively connected to the at least one processor; wherein,

[0039] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the NL2SQL model query enhancement method based on reinforcement learning as described in any of the above examples.

[0040] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured to be: the NL2SQL model query enhancement method based on reinforcement learning described in any of the above examples.

[0041] The NL2SQL model query enhancement method based on reinforcement learning proposed in this application can bring the following beneficial effects:

[0042] By building a reinforcement learning model and forming a query enhancement framework with the NL2SQL model, we achieve efficient and accurate conversion of natural language queries into SQL statements, breaking through the bottleneck of traditional solutions that rely on massive supervisory data and have poor cross-scenario adaptability.

[0043] By adopting a group-relative strategy optimization algorithm and a dynamic reward feedback mechanism, the actual database search results are used as reinforcement learning reward signals through the performance of multi-dimensional indicators, and the large language model is driven through a two-stage structured output of the inference stage and the output stage to autonomously explore query enhancement strategies.

[0044] This improves the accuracy of dynamic matching between user query intent and database structure information while ensuring the stability of model training. Compared to query enhancement methods based on supervised learning, this overcomes the reliance on large-scale supervised data and can achieve a closed-loop training loop using self-generated SQL execution feedback.

[0045] Through technological innovation, we provide a feasible solution for intelligent data retrieval and utilization systems, and promote the in-depth application of new AI technologies in information retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 Schematic diagram of the process of the NL2SQL model query enhancement method based on reinforcement learning in an embodiment of the present application;

[0048] Figure 2 This is a schematic diagram of a NL2SQL model query enhancement method based on reinforcement learning in one scenario in an embodiment of the present application;

[0049] Figure 3 Schematic diagram of the NL2SQL model query enhancement device based on reinforcement learning in an embodiment of the present application. DETAILED DESCRIPTION

[0050] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] The following describes in detail the technical solutions provided by various embodiments of the present application in conjunction with the accompanying drawings.

[0052] like Figure 1 and Figure 2 As shown, the embodiment of the present application provides an NL2SQL model query enhancement method based on reinforcement learning, including:

[0053] S101: Determine a reinforcement learning model, where the reinforcement learning model is used to perform a query enhancement task, where the query enhancement task is used to output a corresponding enhanced query language based on a natural query language input by a user.

[0054] The main function of the reinforcement learning model is to convert the natural query language entered by the user into an enhanced query language more suitable for the query task. The model architecture of the reinforcement learning model can be configured based on actual needs. For example, it can be set to a model based on the Transformer architecture or the Long Short-Term Memory (LSTM) architecture.

[0055] It should be noted that the reinforcement learning model in this application can be used as a separate framework, independent of and in conjunction with the NL2SQL model. It can also be integrated with the NL2SQL model into a complete framework by merging corresponding layers to achieve corresponding statement conversion and task execution.

[0056] Typically, users directly enter queries in natural language, such as "show sales in region A." Zeng Qian's enhanced query language includes not only natural language but also structural information. This improves query accuracy and eliminates the need for supervised data during the entire reinforcement learning training process. For example, the enhanced query language might be "query the amount column in the sales_data table for region A."

[0057] During actual training, context information can be defined based on the database's structural information and integrated to form a corresponding context set. Here, the structural information in the database is defined as context set D. The structural information in this context set is also referred to as context information, and includes database table names, column names, column types, column meanings, primary and foreign key information, and more.

[0058] Here, the input natural query language is called q, and the enhanced query language is called For the NL2SQL model, it can convert the input natural language q into an SQL query statement, execute the SQL query statement, and finally return the corresponding execution result. At this time, the structure information to which it belongs can be determined based on the content contained in the execution result, thereby determining the context information corresponding to the structure information. .

[0059] In the query enhancement task, the original natural query language is converted into an enhanced query language, making the NL2SQL model more efficient and accurate in retrieving relevant context. The query enhancement task of the reinforcement learning model is regarded as a reinforcement learning problem, where the natural query language is set as the state and the enhanced query language is set as the action.

[0060] At this point, for the reinforcement learning model, it can return the enhanced query language related to q based on the natural query language q , at this time, Input into the NL2SQL model, the NL2SQL model is based on Contextual information retrieved , compared to the results obtained directly through q retrieval , which can be more efficient and accurate.

[0061] Establish an end-to-end training framework; the end-to-end training framework includes a group-relative policy optimization algorithm and a reward function. The group-relative policy optimization algorithm generates reasoning steps in the inference phase and generates an enhanced query language in the output phase; the end-to-end training framework performs iterative training based on the reward function.

[0062] End-to-end training is a training method in machine learning that directly models and optimizes the entire process from input data to output without relying on manually designed intermediate steps or feature engineering. During the training of the reinforcement learning model, the reinforcement learning model is used to complete the process from the original input q to the final output. All computations, all intermediate features and representations are automatically learned by the model, thus reducing labor costs.

[0063] Establish an end-to-end training framework to enable the model to autonomously iterate based on database retrieval reward signals, improving the semantic alignment ability of natural language and SQL.

[0064] The relative strategy optimization algorithm and reward function will be explained in detail when they are involved below.

[0065] Establish query scenarios; there are multiple query scenarios, and the database structure information in different query scenarios is different, which is used to test the universality of the reinforcement learning model.

[0066] After training the reinforcement learning model, you can set up different query scenarios to verify the versatility of the reinforcement learning model. Different databases contain different structural information, and the scenarios faced by the reinforcement learning model are also different. Through this test, the model parameters in the reinforcement learning model can be iteratively fine-tuned to increase the versatility and general applicability of the reinforcement learning model.

[0067] Specifically, during the training of the reinforcement learning model, multiple query scenarios can be pre-set. When training the model for a single query scenario, the most common context information corresponding to each execution result in the current query scenario is determined.

[0068] At this time, alternative structural information can be generated based on the most common context information. For example, the table name, column name, etc. in the context information can be replaced with synonyms, or the context information can be deleted and replaced with columns representing other meanings.

[0069] At this time, new query scenarios are generated for training the reinforcement learning model, so that the model can feedback more uncommon contextual information and increase the versatility of the model.

[0070] S102: Acquire multiple existing natural query languages, convert the natural query languages into SQL query statements through the NL2SQL model and execute them, determine the context information corresponding to the structural information and the performance of the context information in pre-set multi-dimensional indicators based on the execution results, so as to generate training samples.

[0071] Reinforcement learning is used to train a reinforcement learning model to enhance the query capability of the NL2SQL model, avoiding the consumption of expensive computing resources to generate supervised data.

[0072] For the preparation of reinforcement learning training data sets, a large number of user natural language query statements can be collected. These queries can come from different application scenarios and different user groups, covering various types such as simple queries and complex queries to ensure the diversity of the data set.

[0073] As mentioned above, the natural query language q is used as input, converted into an SQL query statement through the NL2SQL model, and the SQL query statement is executed, and the corresponding execution result is finally returned. At this time, the structure information to which it belongs can be determined based on the content contained in the execution result, thereby determining the context information corresponding to the structure information. .

[0074] At this point, you can also obtain the execution results to analyze the performance of multiple indicators. For example, these indicators include query accuracy, recall, F1 index, execution efficiency, and whether errors are reported.

[0075] Of course, to improve the quality and reliability of the data set, the data can be cleaned and filtered to remove invalid queries (for example, queries that cannot be executed due to syntax errors) and abnormal retrieval results (for example, incorrect results caused by database failures).

[0076] At this point, a reinforcement learning dataset containing a large number of user queries, candidate SQL statements, database structure information, and performance indicators can be constructed to provide sufficient training data for the subsequent reinforcement learning model training framework, ensuring that the model can effectively learn based on retrieval results in different scenarios.

[0077] S103: Based on a group relative strategy optimization algorithm, the reinforcement learning model is trained through the training samples according to a preset reward function; the reward function is set based on the performance of multi-dimensional indicators of the context information corresponding to the natural query language and the enhanced query language respectively.

[0078] In the previous section, we've already obtained the performance of the original natural query language based on the training samples. We can assume this represents the performance corresponding to the actual contextual information. At this point, the reinforcement learning model generates enhanced query language based on the natural query language. Following the path described above, it generates and executes SQL queries, and the performance corresponding to the enhanced query language is determined based on the execution results.

[0079] When the performance is worse than that corresponding to the actual context, a reward is given according to the reward function, providing feedback to guide the optimization algorithm to adjust the model parameters. If the performance is worse than that corresponding to the actual context, no reward is given, or a negative reward is provided as feedback.

[0080] Specifically, the reward function includes a general part and a separate part. The general part includes multiple first indicators for evaluating the performance of natural query language and enhanced query language in the first indicators; the separate part includes multiple second indicators for evaluating the performance of enhanced query language in the second indicators; the first indicators include accuracy, recall rate, and F1 indicator, and the second indicators include query time and whether an error is reported; the second indicators are set with corresponding weight factors.

[0081] The reward function can be expressed as formula 1:

[0082] Formula 1;

[0083] in, is a natural query language, To enhance the query language, Indicates the reward earned by the large language model generating the corresponding SQL statement for the query statement and performing the search in the database; Context information corresponding to natural query language; To enhance the context information corresponding to the query language; Indicates the accuracy of the query information, which is used to determine whether the search results accurately meet user needs; Indicates the recall rate of the query information, which is used to determine whether the required data is fully obtained; The F1 index represents the query information, which is used to represent the comprehensive precision and recall rate. Indicates query time, used to indicate query efficiency; Indicates whether the query reports an error; a and b are both constant weight factors used to balance indicator weights.

[0084] Utilizing actual database retrieval results as reward signals enables the reinforcement learning model to effectively learn query generation strategies through trial-and-error behavior.

[0085] Specifically, we use the Group Relative Policy Optimization (GRPO) reinforcement learning algorithm for training optimization. The reinforcement learning model is trained to first <think>part) generates the reasoning steps, and then in the output stage (corresponding to <answer>part) to generate the final enhanced query language.

[0086] This structured reasoning mode implements explicit thought-chain reasoning, enabling the reinforcement learning model to consider various aspects of the query and explore different query enhancement strategies before inferring the final answer.

[0087] Among them, the GRPO algorithm design is shown in Formula 2:

[0088] Formula 2;

[0089] in, As the objective function, by optimizing the strategy parameters , maximize the objective function; E represents the mathematical expectation, G represents the number of groups, Indicates the execution result of the SQL query statement of group i, that is, the query generation result of group i. express The corresponding quantity, Indicates limiting the range of change. The scope is limited to , represents the reward function, The KL regularization term is used to limit the query generation difference to not be too large to prevent training collapse. , indicating the new strategy and reference strategies In the generation The probability ratio when Indicates taking the minimum value of the two.

[0090] Furthermore, when training the model through a reward function, the reward function can be set as a dynamic layered function to train the model step by step so that the model prioritizes the most basic indicators such as accuracy, and then gradually optimizes other indicators such as recall rate and performance.

[0091] During the training process, based on the general portion of the reward function, at least a portion of the first indicator is selected as the current indicator, and based on the current indicator, the reinforcement learning model is trained using training samples until training expectations are met. The training expectations include reaching a preset number of training rounds and / or the performance of the enhanced query language output by the reinforcement learning model meeting a preset standard.

[0092] Generally speaking, users can tolerate fewer retrieved items, but they often don't tolerate errors in the retrieved content. Therefore, the first metric to choose is accuracy. Compared to other first metrics, it is more fundamental and ensures the most basic accuracy during the initial model training process. In this case, the reward function can only include the dimension related to accuracy, temporarily omitting other dimensions.

[0093] Among the remaining first indicators, continue to select at least some of the first indicators, add them to the current indicators, and train the reinforcement learning model based on the added current indicators until all the first indicators are added to the current indicators and the training expectations are met.

[0094] After basic accuracy training is complete and training expectations are met, recall and F1 metrics can be added sequentially to continue model training. At this point, the reward function changes from initially including only accuracy to including both accuracy and recall, and then to including both accuracy, recall, and F1. By continuously optimizing the reward function and adding new primary metrics, the reinforcement learning model can gradually train recall and F1 metrics based on the basic accuracy training.

[0095] The second indicator is added to the current indicator, and based on the training process of the first indicator, the weight factor corresponding to the second indicator is set, and the reinforcement learning model is trained according to the current indicator.

[0096] The first metric can be used to evaluate the performance of both natural query language and enhanced query language, so training the first metric is prioritized. After training the first metric, the second metric is added to the "current metric" to provide additional rewards for enhanced query language.

[0097] Of course, in a practical situation, you can also choose to add the second indicator "whether an error is reported" as a training target with the same priority as the accuracy rate to the initial "current indicator" to prioritize training on the basic grammatical issues of the enhanced query language.

[0098] When setting the weight factor corresponding to the second indicator, the model state of the current reinforcement learning model can be clarified based on the training process of the first indicator, and the corresponding weight factor can be set.

[0099] For each round of training of the first indicator, a weighted sum is performed according to the corresponding single-round training efficiency and the corresponding single-round performance to obtain the corresponding overall training efficiency and overall performance; wherein, when the current indicator changes and enters the next round of training, the weight of the later training process is higher.

[0100] Taking the example above as an example, when the current metric only includes accuracy, determine the single-round training efficiency and corresponding single-round performance at the completion of training (that is, when training expectations are met). The single-round training efficiency here can be determined by the number of training rounds. The higher the number of training rounds, the lower the single-round training efficiency. In this case, the single-round performance only includes accuracy.

[0101] After adding recall to the current metrics, a new training process begins and the single-round training efficiency and corresponding single-round performance are calculated. The calculation method for single-round training efficiency remains unchanged, while single-round performance includes precision and recall.

[0102] After adding the F1 indicator to the current indicator, a new training process is entered again. At this time, the calculation method of single-round training efficiency remains unchanged, and the single-round performance includes accuracy, recall rate, and F1 indicator.

[0103] At this time, the overall training efficiency is obtained by weighted summation based on the three single-round training efficiencies, and the various dimensional indicators in the overall performance are also obtained by weighted summation. For example, for the overall training efficiency, the weights corresponding to the three rounds of training processes are 0.2, 0.3, and 0.5, respectively, and the weighted summation is used to obtain the overall training efficiency. As for the overall performance, the weights corresponding to the three rounds of training processes can be set more discretely to better highlight the state of the final model; among them, for the accuracy, since it appears in the three rounds of training, the weights can be set to 0.1, 0.3, and 0.6; and for the recall rate, since it only appears in the second and third rounds of training, its weights can be set to 0, 0.3, and 0.7; for the F1 indicator, since it only appears in the third round of training, its weight is set to 0, 0, and 1. In this way, for each dimensional indicator, the performance of the model for the dimensional indicator during the training process of the first indicator can be obtained by weighting.

[0104] At this time, the weight factor corresponding to the second indicator is determined based on the overall training efficiency and overall performance; wherein, the higher the overall training efficiency and the better the overall performance, the higher the weight factor corresponding to the second indicator. For example, a corresponding positive correlation mapping relationship is set between the overall training efficiency and the overall performance and each weight factor corresponding to the second indicator, and the weight of the overall training efficiency and the overall performance is set for each weight factor. In this way, the weight factor of the second indicator can be calculated based on the overall training efficiency, the overall performance, and the weight of each in each weight factor.

[0105] Higher overall training efficiency and better overall performance indicate that the model performs better during the training of the first metric. In this case, more attention can be paid to training the weights of the enhanced query language to increase the overall accuracy of the model. If the overall training efficiency or overall performance is poor, set the weight factor for the second metric lower, allowing more attention to the basic content related to the first metric during the training of the second metric.

[0106] By building a reinforcement learning model and forming a query enhancement framework with the NL2SQL model, we achieve efficient and accurate conversion of natural language queries into SQL statements, breaking through the bottleneck of traditional solutions that rely on massive supervisory data and have poor cross-scenario adaptability.

[0107] By adopting a group-relative strategy optimization algorithm and a dynamic reward feedback mechanism, the actual database search results are used as reinforcement learning reward signals through the performance of multi-dimensional indicators, and the large language model is driven through a two-stage structured output of the inference stage and the output stage to autonomously explore query enhancement strategies.

[0108] This improves the accuracy of dynamic matching between user query intent and database structure information while ensuring the stability of model training. Compared to query enhancement methods based on supervised learning, this overcomes the reliance on large-scale supervised data and can achieve a closed-loop training loop using self-generated SQL execution feedback.

[0109] Through technological innovation, we provide a feasible solution for intelligent data retrieval and utilization systems, and promote the in-depth application of new AI technologies in information retrieval.

[0110] In one embodiment, during the training of the reinforcement learning model, there may be some situations where the natural query language input by the user is too complex, resulting in too many restrictions in the output enhanced query language, which in turn results in too few or even zero search results returned in the execution results, which is not conducive to the user experience.

[0111] Based on this, if it is determined that the length of the structural enhancement information in the enhanced query language is longer than the preset length, and the performance of the corresponding specified dimension indicator in the multi-dimensional indicators according to the execution result obtained by the enhanced query language is lower than the preset standard, then the corresponding specified keyword in the natural query language is determined based on the structural enhancement information in the enhanced query language.

[0112] Structural enhancement information refers to a statement describing structural information, derived by converting keywords from natural query language. For example, if the natural query is "show sales in region A," and the enhanced query is "query the amount column in the sales_data table for region A," then the "sales_data table" and "amount column" constitute the structural enhancement information. The length of the structural enhancement information can be determined by its number. A designated dimension metric can be recall rate, which indicates that the number of retrieved results is lower than a preset threshold.

[0113] The specified keywords corresponding to the structural enhancement information can be obtained through the GRPO algorithm <think>In this part, the explicit process of the reasoning steps is obtained, and in this explicit chain of thought reasoning, which keywords are included to convert the structural enhancement information.

[0114] At this point, the large language model classifies the specified keywords, identifying the core keywords and flexible keywords they contain. Natural query language and all identified keywords are input, and the large language model classifies each keyword, identifying the core keywords and flexible keywords. Flexible keywords refer to keywords other than core keywords.

[0115] At this point, the structural enhancement information corresponding to the core keywords is retained, and the structural enhancement information corresponding to at least some of the elastic keywords is converted back to the corresponding elastic keywords, resulting in an updated enhanced query language. Specifically, during the execution of the reinforcement learning model's task, only the core keywords and a maximum of some of the elastic keywords are converted into structural enhancement information, while the "at least some of the elastic keywords" mentioned above are retained as keywords without undergoing structural enhancement information conversion. At this point, the reinforcement learning model can be used to reverse-restore this "at least some of the elastic keywords," or the forward conversion can be performed by ignoring this "at least some of the elastic keywords" and re-executing the conversion task.

[0116] After obtaining the updated enhanced query language, it can be used as an output of the current natural query language to determine the performance of the output based on the reward function and to guide the parameters of the reinforcement learning model.

[0117] Furthermore, when restoring or reconverting at least part of the elastic keywords, among all the elastic keywords, a corresponding number of elastic keywords are selected as keyword combinations based on the selection quantity corresponding to the current training batch, and the structural enhancement information corresponding to the keyword combination is restored to the corresponding elastic keywords to obtain the updated enhanced query language and its corresponding performance.

[0118] The number of elastic keywords selected for restoration in each training batch varies. For example, if there are currently five elastic keywords, four will be selected for restoration in the first training batch (this results in four selection methods), three in the second training batch (this results in ten selection methods), and two in the third training batch (this results in ten selection methods).

[0119] Here, each selection method is regarded as a keyword combination under the training batch, thereby obtaining the updated enhanced query language corresponding to each keyword combination and obtaining the corresponding performance.

[0120] For all keyword combinations in each training batch, the updated enhanced query language corresponding to several keyword combinations with the best performance of specified dimension indicators is used to calculate the corresponding reward function and used to train the reinforcement learning model; the training progress of the reinforcement learning model is positively correlated with the number of selected keyword combinations.

[0121] To focus resources on optimizing high-return strategies and improve training efficiency, we don't calculate a reward function for every selection method (that is, every keyword combination). Instead, we select the keyword combinations that perform best on a specific metric (e.g., recall) within each training batch. We calculate the reward function for these enhanced query languages and use it to train the reinforcement learning model. This allows the reinforcement learning model to selectively transform keywords in complex natural query languages, providing a certain level of recall.

[0122] As the reinforcement learning model progresses, that is, at the very beginning of training, the focus should be on the most common choices, so a smaller number of keyword combinations are selected. For example, only one keyword combination with the best performance is selected for each batch. As the reinforcement learning model progresses and nears completion, targeted training should be performed to address the complexities of natural query language. Therefore, a larger number of keyword combinations can be selected to calculate the reward function for model training.

[0123] like Figure 3 As shown, the embodiment of the present application also provides an NL2SQL model query enhancement device based on reinforcement learning, including:

[0124] at least one processor; and,

[0125] a memory communicatively connected to the at least one processor; wherein,

[0126] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the NL2SQL model query enhancement method based on reinforcement learning as described in any of the above embodiments.

[0127] An embodiment of the present application further provides a non-volatile computer storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to be: the NL2SQL model query enhancement method based on reinforcement learning described in any of the above embodiments.

[0128] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the device and medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0129] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects to their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0130] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.< / think> < / answer> < / think>

Claims

1. A NL2SQL model query enhancement method based on reinforcement learning, characterized in that: include: Determining a reinforcement learning model, wherein the reinforcement learning model is used to perform a query enhancement task, wherein the query enhancement task is used to output corresponding enhanced query language based on a natural query language input by a user; Obtain multiple existing natural query languages, convert the natural query languages into SQL query statements using the NL2SQL model, and execute the SQL query statements. Based on the execution results, determine the context information corresponding to the structural information and the performance of the context information in pre-set multi-dimensional indicators to generate training samples. Based on a group relative strategy optimization algorithm, the reinforcement learning model is trained using the training samples according to a preset reward function; the reward function is set based on the performance of multi-dimensional indicators of context information corresponding to the natural query language and the enhanced query language respectively; Before obtaining the training samples generated according to the natural query language, the method further includes: Define context information based on the database structure information and integrate it to obtain the corresponding context set; Define the query enhancement task of the reinforcement learning model, set the natural query language as the state and the enhanced query language as the action; Establishing an end-to-end training framework; the end-to-end training framework includes a group relative policy optimization algorithm and a reward function, the group relative policy optimization algorithm generates reasoning steps in the inference phase and generates an enhanced query language in the output phase; the end-to-end training framework performs iterative training based on the reward function; Establishing a query scenario; there are multiple query scenarios, and the structural information of the database in different query scenarios is different, which is used to test the universality of the reinforcement learning model.

2. The NL2SQL model query enhancement method based on reinforcement learning according to claim 1, characterized in that: The reward function includes a general part and a separate part, wherein the general part includes a plurality of first indicators for evaluating the performance of the natural query language and the enhanced query language in terms of the first indicators; The separate part includes a plurality of second indicators for evaluating the performance of the enhanced query language in the second indicators; The first indicators include accuracy, recall rate, and F1 index, and the second indicators include query time and whether an error is reported; The second indicator is provided with a corresponding weight factor.

3. The NL2SQL model query enhancement method based on reinforcement learning according to claim 2, characterized in that: Based on the group relative strategy optimization algorithm and according to the preset reward function, the reinforcement learning model is trained through the training samples, specifically including: Determine the corresponding reward function based on the group relative strategy optimization algorithm; Based on the general portion of the reward function, at least part of the first indicator is selected as a current indicator, and based on the current indicator, the reinforcement learning model is trained using the training samples until a training expectation is met; the training expectation includes a predetermined number of training rounds and / or a performance of an enhanced query language output by the reinforcement learning model meets a predetermined standard; Continue to select at least some of the remaining first indicators, add them to the current indicators, and train the reinforcement learning model based on the added current indicators until all the first indicators are added to the current indicators and the training expectations are met; The second indicator is added to the current indicator, and based on the training process of the first indicator, a weight factor corresponding to the second indicator is set, and the reinforcement learning model is trained according to the current indicator.

4. The NL2SQL model query enhancement method based on reinforcement learning according to claim 3 is characterized in that: Based on the training process of the first indicator, a weight factor corresponding to the second indicator is set, specifically including: For each round of training of the first indicator, weighted summation is performed based on the corresponding single-round training efficiency and the corresponding single-round performance to obtain the corresponding overall training efficiency and overall performance; wherein, when the current indicator changes, the next round of training begins, and the later the training process, the higher the weight; A weight factor corresponding to the second indicator is determined based on the overall training efficiency and the overall performance; wherein, the higher the overall training efficiency and the better the overall performance, the higher the weight factor corresponding to the second indicator.

5. The NL2SQL model query enhancement method based on reinforcement learning according to claim 1, characterized in that: The method further comprises: During the training process of the reinforcement learning model, if it is determined that the length of the structural enhancement information in the enhanced query language is longer than a preset length, and the performance of a specified dimension indicator in the corresponding multi-dimensional indicators according to the execution result obtained by the enhanced query language is lower than a preset standard, then determining the specified keyword corresponding to the structural enhancement information in the natural query language according to the structural enhancement information in the enhanced query language; Using a large language model, the designated keywords are graded to determine the core keywords and flexible keywords contained therein; The structural enhancement information corresponding to the core keywords is retained, and the structural enhancement information corresponding to at least part of the elastic keywords is restored to the corresponding elastic keywords, so as to obtain an updated enhanced query language.

6. The NL2SQL model query enhancement method based on reinforcement learning according to claim 5, characterized in that: Restoring the structural enhancement information corresponding to at least some of the elastic keywords to the corresponding elastic keywords to obtain an updated enhanced query language specifically includes: Among all the elastic keywords, a corresponding number of elastic keywords are selected as keyword combinations according to the number of selections corresponding to the current training batch, and the structural enhancement information corresponding to the keyword combinations is restored to the corresponding elastic keywords to obtain an updated enhanced query language and obtain its corresponding performance; For all keyword combinations in each training batch, the updated enhanced query language corresponding to several keyword combinations with the best performance of the specified dimension indicators is used to calculate the corresponding reward function and used to train the reinforcement learning model; the training progress of the reinforcement learning model is positively correlated with the number of selected keyword combinations.

7. The NL2SQL model query enhancement method based on reinforcement learning according to claim 1, characterized in that: Establish a query scenario, including: During the training of the reinforcement learning model, determining the most common context information corresponding to each execution result in the current query scenario; Based on the most common pieces of context information, alternative structural information is generated for substitution to generate a new query scenario for training the reinforcement learning model.

8. A NL2SQL model query enhancement device based on reinforcement learning, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the NL2SQL model query enhancement method based on reinforcement learning as described in any one of claims 1 to 7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured as: the NL2SQL model query enhancement method based on reinforcement learning as described in any one of claims 1 to 7.

Images