NL2SQL model query enhancement method and device based on reinforcement learning and medium

Abstract

The invention discloses an NL2SQL model query enhancement method and device based on reinforcement learning and a medium, and relates to the field of data query, the method comprises the steps of obtaining a plurality of existing natural query languages, converting the natural query languages into SQL query statements through an NL2SQL model and executing the SQL query statements, determining context information corresponding to structure information according to an execution result, according to the context information and the performance of the context information in a preset multi-dimensional index, generating a training sample; based on a group relative strategy optimization algorithm, training the reinforcement learning model through the training sample according to a preset reward function; the reward function is based on performance expression setting in the multi-dimensional indexes of the context information corresponding to the natural query language and the enhanced query language respectively. The reinforcement learning model and the NL2SQL model form a query enhancement framework, so that efficient and accurate conversion from natural language query to SQL statements is realized, and the bottlenecks of dependence on massive supervision data and poor cross-scene adaptability are broken through.

Description

Technical Field

[0001] This application relates to the field of data query, and particularly to a method, device, and medium for query enhancement of an NL2SQL model based on reinforcement learning. Background Art

[0002] Currently, information retrieval technology based on large language models (LLMs) has been applied in the field of data retrieval. In terms of structured data retrieval, it is embodied in the use of a natural language to structured query language (NL2SQL) model to identify the intent of a user's natural language query and generate a corresponding structured query language (SQL) statement, associating the large language model with a relational database to collaboratively retrieve relevant data information.

[0003] However, in traditional solutions, the retrieval quality of the NL2SQL model is challenged by many factors, mainly including: the ambiguity of natural language queries, complex database schemas, semantic alignment problems, insufficient training data, etc.

[0004] For the above problems, the retrieval quality of the NL2SQL model can be improved through retrieval-augmented generation (RAG) technology. Retrieval augmentation uses a large language model to expand or rewrite user queries to better capture relevant information. By bridging the semantic gap between the user's information needs and the document collection, retrieval augmentation helps retrieve more relevant content.

[0005] However, in traditional solutions, query enhancement methods based on large language models usually adopt supervised learning or distillation techniques, which require a large amount of computing resources to generate supervised data, and the model training quality depends to a large extent on the quality and coverage of the supervised data, resulting in uncontrollable model quality. Summary of the Invention

[0006] To solve the above problems, this application proposes a method for query enhancement of an NL2SQL model based on reinforcement learning, including: Determine a reinforcement learning model, where the reinforcement learning model is used to perform a query enhancement task, and the query enhancement task is used to output a corresponding enhanced query language according to the natural query language input by the user; Obtain multiple existing natural query languages, convert them into SQL query statements through the NL2SQL model and execute them for the natural query languages, determine the context information corresponding to the structural information according to the execution results, and the performance of the context information in a pre-set multi-dimensional index, so as to generate training samples; Based on the group relative policy optimization algorithm, the reinforcement learning model is trained with the training samples according to a pre-set reward function; the reward function is set based on the performance in multi-dimensional metrics of the context information corresponding to the natural query language and the enhanced query language respectively.

[0007] In one example, before obtaining the training samples generated according to the natural query language, the method further includes: Define the context information according to the structure information of the database, and integrate to obtain the corresponding context set; Define the query enhancement task of the reinforcement learning model, and set the natural query language as the state and the enhanced query language as the action; Establish an end-to-end training framework; the end-to-end training framework includes a group relative policy optimization algorithm and a reward function, the group relative policy optimization algorithm generates inference steps in the inference stage and generates an enhanced query language in the output stage; the end-to-end training framework is iteratively trained based on the reward function; Establish query scenarios; there are multiple query scenarios, and the structure information of the database in different query scenarios is different, which is used for the generality test of the reinforcement learning model.

[0008] In one example, the reward function includes a general part and a separate part, the general part includes multiple first metrics, which are used to evaluate the performance of the natural query language and the enhanced query language in the first metrics; the separate part includes multiple second metrics, which are used to evaluate the performance of the enhanced query language in the second metrics; The first metrics include accuracy, recall, and F1 metric, and the second metrics include query time and whether there is an error; The second metrics are set with corresponding weight factors.

[0009] In one example, based on the group relative policy optimization algorithm, the reinforcement learning model is trained with the training samples according to a pre-set reward function, which specifically includes: Based on the group relative policy optimization algorithm, determine the corresponding reward function; Based on the general part of the reward function, select at least some of the first metrics as the current metrics, and based on the current metrics, train the reinforcement learning model with the training samples until the training expectation is met; the training expectation includes that the number of training rounds reaches a preset number of rounds, and / or, the performance of the enhanced query language output by the reinforcement learning model reaches a preset standard; Among the remaining first metrics, continue to select at least some of the first metrics, add them to the current metrics, and train the reinforcement learning model based on the current metrics after addition until all the first metrics are added to the current metrics and the training expectation is achieved; Add the second metric to the current metrics, set the weight factor corresponding to the second metric based on the training process of the first metric, and train the reinforcement learning model according to the current metrics.

[0010] In one example, setting the weight factor corresponding to the second metric based on the training process of the first metric specifically includes: For each round of training process of the first metric, perform weighted summation according to its corresponding single-round training efficiency and corresponding single-round performance to obtain the corresponding overall training efficiency and overall performance; wherein, when the current metrics change, enter the next round of training process, and the weights of the later training processes are higher; Determine the weight factor corresponding to the second metric according to the overall training efficiency and the overall performance; wherein, the higher the overall training efficiency and the better the overall performance, the higher the weight factor corresponding to the second metric.

[0011] In one example, the method further includes: During the training process of the reinforcement learning model, if it is determined that the length of the structural enhancement information in the enhanced query language is higher than the preset length, and the performance of the specified dimension metric in the corresponding multi-dimensional metrics of the execution result obtained according to the enhanced query language is lower than the preset standard, then determine the specified keyword corresponding to it in the natural query language according to the structural enhancement information in the enhanced query language; Perform hierarchical processing on the specified keyword through a large language model to determine the core keyword and flexible keyword contained therein; Retain the structural enhancement information corresponding to the core keyword, and restore the structural enhancement information corresponding to at least some of the flexible keywords to the corresponding flexible keywords to obtain the updated enhanced query language.

[0012] In one example, restoring the structural enhancement information corresponding to at least some of the flexible keywords to the corresponding flexible keywords to obtain the updated enhanced query language specifically includes: Among all the flexible keywords, select the corresponding number of flexible keywords as a keyword combination according to the selection quantity corresponding to the current training batch, restore the structural enhancement information corresponding to the keyword combination to the corresponding flexible keywords to obtain the updated enhanced query language, and obtain its corresponding performance. For all keyword combinations in each training batch, the updated enhanced query languages corresponding to several keyword combinations with the optimal performance of the specified dimensional indicators are used to calculate the corresponding reward function and used for training the reinforcement learning model; the training progress of the reinforcement learning model is positively correlated with the number of selected keyword combinations.

[0013] In one example, a query scenario is established, specifically including: During the training process of the reinforcement learning model, determine the most common several context information corresponding to each execution result in the current query scenario; Generate replacement structure information for replacement according to the most common several context information to generate a new query scenario for training the reinforcement learning model.

[0014] On the other hand, the present application also proposes a query enhancement device for an NL2SQL model based on reinforcement learning, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the query enhancement method for the NL2SQL model based on reinforcement learning as described in any of the above examples.

[0015] On the other hand, the present application also proposes a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set as: the query enhancement method for the NL2SQL model based on reinforcement learning as described in any of the above examples.

[0016] The query enhancement method for the NL2SQL model based on reinforcement learning proposed by the present application can bring the following beneficial effects: By constructing a reinforcement learning model and forming a query enhancement framework with the NL2SQL model, efficient and accurate conversion from natural language queries to SQL statements is achieved, breaking through the bottlenecks of relying on a large amount of supervised data and poor cross-scenario adaptability in traditional solutions.

[0017] Adopt the group relative policy optimization algorithm and the dynamic reward feedback mechanism, use the performance of multi-dimensional indicators of the actual database retrieval results as the reinforcement learning reward signal, and drive the large language model to autonomously explore the query enhancement strategy through the two-stage structured output of the inference stage and the output stage.

[0018] It improves the dynamic matching accuracy between the user's query intention and the database structure information, while ensuring the stability of model training. Compared with the query enhancement method based on supervised learning, it overcomes the dependence on large-scale supervised data and can utilize the self-generated SQL execution feedback to achieve a training closed-loop.

[0019] Through technological innovation, it provides a feasible solution for the intelligent data retrieval and utilization system, promoting the in-depth application of new AI technologies in information retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The drawings described herein are used to provide a further understanding of the present application and form a part of the present application. The illustrative embodiments and descriptions thereof of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the drawings: Figure 1 It is a schematic flow chart of the query enhancement method for the NL2SQL model based on reinforcement learning in the embodiments of the present application; Figure 2 It is a schematic diagram of the query enhancement method for the NL2SQL model based on reinforcement learning in a certain situation in the embodiments of the present application; Figure 3 It is a schematic diagram of the query enhancement device for the NL2SQL model based on reinforcement learning in the embodiments of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] To make the objectives, technical solutions, and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0022] The following will detail the technical solutions provided by each embodiment of the present application in conjunction with the drawings.

[0023] As Figure 1 and Figure 2 shown, the embodiments of the present application provide a query enhancement method for the NL2SQL model based on reinforcement learning, including: S101: Determine a reinforcement learning model, where the reinforcement learning model is used to perform a query enhancement task, and the query enhancement task is used to output a corresponding enhanced query language according to the natural query language input by the user.

[0024] The main function of the reinforcement learning model is to convert the natural query language input by the user into an enhanced query language that is more suitable for query tasks. The model architecture of the reinforcement learning model can be set accordingly based on actual needs. For example, it can be set as a model based on the Transformer architecture or the Long Short-Term Memory (LSTM) architecture.

[0025] It should be noted that the reinforcement learning model in this application can be used as an independent framework, independent of and cooperating with the NL2SQL model. It can also be integrated with the NL2SQL model into a complete framework through the merger of corresponding layers, so as to achieve the corresponding statement conversion and task execution.

[0026] Generally speaking, the languages directly input by users for queries are all natural query languages. For example, it can be "Show the sales amount in area A". In the enhanced query language, it can not only include natural language, but also include corresponding structural information, so that when querying through this enhanced query language, the query accuracy can be increased, and the entire reinforcement learning training process does not require the setting of supervised data. For example, the enhanced query language can be "Query the amount column in the sales_data table in area A".

[0027] In the actual training process, the context information can also be defined according to the structural information of the database and integrated to obtain the corresponding context set. Here, the structural information in the database is defined as the context set D, and the structural information in this context set is also called context information, which includes database table names, column names, column types, column meanings, primary and foreign key information, etc.

[0028] Here, the input natural query language is called q, and the enhanced query language is called . For the NL2SQL model, it can convert the input natural query language q into an SQL query statement, execute this SQL query statement, and finally return the corresponding execution result. At this time, according to the content contained in this execution result, the corresponding structural information can be determined, so as to determine the context information corresponding to this structural information .

[0029] In the query enhancement task, the original natural query language is converted into an enhanced query language, making the NL2SQL model more efficient and accurate in retrieving relevant contexts. The query enhancement task of the reinforcement learning model is regarded as a reinforcement learning problem, where the natural query language is set as the state and the enhanced query language is set as the action.

[0030] At this time, for the reinforcement learning model, it can return the enhanced query language related to q according to the natural query language q At this time, is input into the NL2SQL model, and the NL2SQL model retrieves according to the context information obtained by retrieval , compared with the retrieved directly through q, it can be more efficient and accurate.

[0031] An end-to-end training framework is established; the end-to-end training framework includes a group relative policy optimization algorithm and a reward function. The group relative policy optimization algorithm generates inference steps in the inference stage and generates an enhanced query language in the output stage; the end-to-end training framework performs iterative training based on the reward function.

[0032] End-to-End Training is a training method in machine learning, which refers to directly modeling and optimizing from input data to output results without relying on manually designed intermediate steps or feature engineering. During the training process of the reinforcement learning model, through this reinforcement learning model, all calculations from the original input q to the final output are completed, and all intermediate features and representations are automatically learned by the model, thereby reducing labor costs.

[0033] An end-to-end training framework is established to enable the model to autonomously iterate based on the database retrieval reward signal and improve the semantic alignment ability between natural language and SQL.

[0034] The relative policy optimization algorithm and the reward function will be explained in detail when they are involved below.

[0035] Query scenarios are established; there are multiple query scenarios, and the structure information of the databases in different query scenarios is different, which is used for the generality test of the reinforcement learning model.

[0036] After the reinforcement learning model is trained, different query scenarios can be set to verify the generality of the reinforcement learning model. The structure information in different databases is different. For the reinforcement learning model, the scenarios it faces are also different. Through this test, the model parameters in the reinforcement learning model are iteratively fine-tuned, thereby increasing the generality and versatility of the reinforcement model.

[0037] Specifically, during the training process of the reinforcement learning model, multiple query scenarios can be preset in advance. When training the model for a single query scenario, determine the most common several context information corresponding to each execution result in the current query scenario.

[0038] At this time, based on the most common several pieces of context information, alternative structure information can be generated for substitution. For example, synonyms of table names, column names, etc. in the context information can be replaced, or the context information can be deleted and replaced with columns representing other meanings.

[0039] At this time, a new query scenario can be generated for training the reinforcement learning model, enabling the model to provide more feedback on uncommon context information and increasing the versatility of the model.

[0040] S102: Obtain multiple existing natural query languages, convert them into SQL query statements through the NL2SQL model and execute them, determine the context information corresponding to the structure information according to the execution results, and the performance of the context information in pre-set multi-dimensional metrics, so as to generate training samples.

[0041] Use reinforcement learning to train the reinforcement learning model to enhance the query ability of the NL2SQL model, so as to avoid consuming expensive computing resources to generate supervised data.

[0042] For the preparation of the reinforcement learning training dataset, a large number of user natural language query statements can be collected. These queries can come from different application scenarios and different user groups, covering various types such as simple queries and complex queries to ensure the diversity of the dataset.

[0043] As mentioned above, taking the natural query language q as the input, converting it into an SQL query statement through the NL2SQL model, executing the SQL query statement, and finally returning the corresponding execution result. At this time, according to the content included in the execution result, the structure information to which it belongs can be determined, and thus the context information corresponding to the structure information can be determined. 。

[0044] At this time, the performance of the execution result in multi-dimensional metrics can also be obtained. For example, the multi-dimensional metrics include query accuracy, recall rate, F1 metric, execution efficiency, and whether there is an error, etc.

[0045] Of course, to improve the quality and reliability of the dataset, the data can be cleaned and filtered to remove invalid queries (such as queries with syntax errors that cannot be executed) and abnormal retrieval results (such as error results caused by database failures).

[0046] At this time, a reinforcement learning dataset containing a large number of user queries, candidate SQL statements, database structure information, and performance can be constructed to provide sufficient training data for the subsequent training framework of the reinforcement learning model, ensuring that the model can effectively learn according to the retrieval results in different scenarios.

[0047] S103: Based on the group relative policy optimization algorithm, train the reinforcement learning model with the training samples according to a pre-set reward function; the reward function is set based on the performance in multi-dimensional metrics of the context information corresponding to the natural query language and the enhanced query language respectively.

[0048] In the above text, the performance corresponding to the original natural query language can already be obtained based on the training samples. Here, it can be assumed that it is the performance corresponding to the true context information. At this time, the reinforcement learning model generates an enhanced query language according to the natural query language, and follows the path in the above text to generate and execute an SQL query statement, and obtains the performance corresponding to the enhanced query language based on the execution result.

[0049] When the performance is better than the performance corresponding to the true context information, then give a reward according to the reward function, provide a feedback signal to guide the optimization algorithm to adjust the model parameters. If the performance is not as good as the performance corresponding to the true context information, then no reward may be given, or a feedback signal for negative reward may be provided.

[0050] Specifically, the reward function includes a general part and a separate part. The general part includes multiple first metrics, which are used to evaluate the performance of the natural query language and the enhanced query language in the first metrics; the separate part includes multiple second metrics, which are used to evaluate the performance of the enhanced query language in the second metrics; the first metrics include accuracy, recall rate, and F1 metric, and the second metrics include query time and whether there is an error; corresponding weight factors are set for the second metrics.

[0051] The reward function can be as shown in Formula 1: Formula 1; Among them, is the natural query language, is the enhanced query language, represents the reward benefit obtained by the large language model generating the corresponding SQL statement for the query statement and performing retrieval in the database; is the context information corresponding to the natural query language; is the context information corresponding to the enhanced query language; represents the accuracy rate of the query information, which is used to judge whether the retrieval result accurately meets the user's needs; represents the recall rate of the query information, which is used to judge whether all the required data is obtained completely; represents the F1 metric of the query information, which is used to represent the metric that combines accuracy rate and recall rate; represents the query time, which is used to represent the query efficiency; Indicates whether the query reports an error; both a and b are constant weight factors used to balance the index weights.

[0052] Using the actual database retrieval results as the reward signal enables the reinforcement learning model to effectively learn the query generation policy through trial-and-error behavior.

[0053] Specifically, we adopt the Group Relative Policy Optimization (GRPO) reinforcement learning algorithm for training and optimization. The reinforcement learning model is trained to first be in the inference phase (corresponding to <think>Part) Generate inference steps, and then in the output stage (corresponding to <answer>Generate the final enhanced query language.

[0054] This structured reasoning pattern implements explicit chain-of-thought reasoning, enabling the reinforcement learning model to consider various aspects of the query and explore different query enhancement strategies before inferring the final answer.

[0055] Among them, the GRPO algorithm is designed as shown in Equation 2: Equation 2; Among them, is the objective function, which maximizes this objective function by optimizing the policy parameters , E represents taking the mathematical expectation, G represents the number of groups, represents the execution result of the SQL query statement in the i-th group, that is, the query generation result of the i-th group, represents the corresponding quantity, represents the restricted variation range, restricting the range to , represents the reward function, represents the KL regularization term used to restrict the query generation difference from being too large to prevent training collapse, , represents the new policy and the reference policy at the time of generating the probability ratio, represents taking the minimum of the two.

[0056] Furthermore, when training the model through the reward function, the reward function can be set as a dynamic hierarchical function to gradually train the model, enabling the model to first ensure the most basic metrics such as accuracy, and then gradually optimize other metrics such as recall rate and performance.

[0057] During the training process, based on the general part of the reward function, at least some of the first metrics are selected as the current metrics, and based on the current metrics, the reinforcement learning model is trained with training samples until the training expectations are met. Among them, the training expectations include that the number of training rounds reaches the preset number of rounds, and / or the performance of the enhanced query language output by the reinforcement learning model reaches the preset standard.

[0058] Generally speaking, users may allow less retrieved content, but they usually do not allow the retrieved content to be incorrect. Therefore, the first metric selected can be accuracy, which is more fundamental than other first metrics, enabling the model to ensure the most basic accuracy during the initial training process. At this time, the reward function can only include dimensions related to accuracy, and other dimensions can be temporarily removed.

[0059] Among the remaining first metrics, continue to select at least some of the first metrics, add them to the current metrics, and train the reinforcement learning model based on the current metrics after addition until all the first metrics are added to the current metrics and the training expectation is met.

[0060] After the most basic accuracy rate training is completed, that is, when the training expectation is met, the recall rate and F1 metric can be added in sequence for model training. At this time, in the reward function, it starts from only including the accuracy rate at the very beginning, and then successively becomes including the accuracy rate and the recall rate, and including the accuracy rate, the recall rate, and the F1 metric, continuously optimizing the reward function and adding new first metrics, so that the reinforcement learning model can gradually train the recall rate and F1 metric on the basis of achieving the most basic accuracy rate training.

[0061] Add the second metric to the current metrics, set the weight factor corresponding to the second metric based on the training process of the first metric, and train the reinforcement learning model according to the current metrics.

[0062] For the first metric, it can be used to evaluate the performance of both natural query language and enhanced query language at the same time. Therefore, the first metric is preferentially trained. After the training of the first metric is completed, add the second metric to the "current metrics" to achieve additional rewards for the enhanced query language.

[0063] Of course, in a practical situation, based on selection, "whether there is an error" in the second metric can also be used as a training target with the same priority as the accuracy rate and added to the initial "current metrics" to preferentially ensure the training of the basic grammar problems of the enhanced query language.

[0064] When setting the weight factor corresponding to the second metric, the model state of the current reinforcement learning model can be clarified based on the training process of the first metric, and the corresponding weight factor can be set.

[0065] For each round of training process of the first metric, according to its corresponding single-round training efficiency and corresponding single-round performance, weighted summation is performed respectively to obtain the corresponding overall training efficiency and overall performance; among them, when the current metrics change and enter the next round of training process, the weight of the later training process is higher.

[0066] Taking the example in the above text as an example, when the current metrics only include the accuracy rate, determine its single-round training efficiency and corresponding single-round performance when the training is completed (that is, when the training expectation is met). Here, the single-round training efficiency can be determined by the number of training rounds. The higher the number of training rounds, the lower the single-round training efficiency, and the single-round performance only includes the accuracy rate at this time.

[0067] After adding the recall rate to the current metrics, a new training process is entered, and the single-round training efficiency and the corresponding single-round performance of this training process are calculated. The calculation method of the single-round training efficiency remains unchanged, while the single-round performance includes the accuracy rate and the recall rate.

[0068] After adding the F1 metric to the current metrics, a new training process is entered again. At this time, the calculation method of the single-round training efficiency remains unchanged, while the single-round performance includes the accuracy rate, the recall rate, and the F1 metric.

[0069] At this time, the overall training efficiency is obtained by weighted summation based on the three single-round training efficiencies, and the metrics of each dimension in the overall performance are also obtained by weighted summation. For example, for the overall training efficiency, the weights corresponding to the training processes of these three rounds are 0.2, 0.3, and 0.5 respectively, and the overall training efficiency is obtained by weighted summation. For the overall performance, the weights corresponding to the training processes of these three rounds can be set more discretely to highlight the state of the finally obtained model more. Among them, for the accuracy rate, since it appears in the training processes of all three rounds, the weights can be set to 0.1, 0.3, and 0.6; for the recall rate, since it only appears in the training processes of the second and third rounds, its weights can be set to 0, 0.3, and 0.7; for the F1 metric, since it only appears in the training process of the third round, its weight is set to 0, 0, and 1. Thus, for each dimension metric, the performance of the model on this dimension metric in the training process of the first metric can be obtained through weighting.

[0070] At this time, according to the overall training efficiency and the overall performance, the weight factor corresponding to the second metric is determined; among them, the higher the overall training efficiency and the better the overall performance, the higher the weight factor corresponding to the second metric. For example, a positive correlation mapping relationship is set separately between the overall training efficiency, the overall performance and each weight factor corresponding to the second metric, and for each weight factor, the weights of the overall training efficiency and the overall performance are set. Thus, the weight factor of the second metric can be calculated according to the overall training efficiency, the overall performance, and their respective weights in each weight factor.

[0071] The higher the overall training efficiency and the better the overall performance, it indicates that the model performs more excellently in the training process of the first metric. At this time, more attention can be paid to the training weight of the enhanced query language alone, so as to increase the overall accuracy of the model. If the overall training efficiency or the overall performance is poor, the weight factor corresponding to the second metric is set lower, so that in the training process of the second metric, more attention can be paid to the basic content related to the first metric.

[0072] By constructing a reinforcement learning model and forming a query enhancement framework with the NL2SQL model, it realizes the efficient and accurate conversion from natural language queries to SQL statements, breaking through the bottlenecks of relying on a large amount of supervised data and poor cross-scenario adaptability in traditional solutions.

[0073] Adopt the group relative policy optimization algorithm and the dynamic reward feedback mechanism. Use the performance of multi-dimensional metrics of the actual database retrieval results as the reinforcement learning reward signal, and drive the large language model to autonomously explore query enhancement strategies through the two-stage structured output of the inference stage and the output stage.

[0074] It improves the dynamic matching accuracy between the user's query intention and the database structure information, and at the same time ensures the stability of model training. Compared with the query enhancement method based on supervised learning, it overcomes the dependence on large-scale supervised data and can utilize the self-generated SQL execution feedback to achieve a training closed-loop.

[0075] Through technological innovation, it provides a practical solution for intelligent data retrieval and utilization systems, promoting the in-depth application of new AI technologies in information retrieval.

[0076] In one embodiment, during the training process of the reinforcement learning model, there may be some situations where the natural query language input by the user is too complex, resulting in too many restrictions in the output enhanced query language, and thus the number of retrieval results returned in the execution result is too small or even 0, which is not conducive to the user experience.

[0077] Based on this, if it is determined that the length of the structural enhancement information in the enhanced query language is higher than the preset length, and the performance of the specified dimension index in the corresponding multi-dimensional metrics of the execution result obtained according to the enhanced query language is lower than the preset standard, then according to the structural enhancement information in the enhanced query language, determine the corresponding specified keyword in the natural query language.

[0078] Among them, the structural enhancement information refers to the statement used to describe the structural information obtained by converting the keywords in the natural query language. For example, if the natural query language is "Show the sales amount in area A", and the enhanced query language is "Query the amount column in the sales_data table in area A", at this time, "sales_data table" and "amount column" are the structural enhancement information. And the length of the structural enhancement information can be determined by its quantity. The specified dimension index can be the recall rate, which is used to indicate that the number of the finally recalled retrieval results in this time is lower than the preset standard.

[0079] The specified keyword corresponding to the structural enhancement information can be obtained through the <think>In this part, for the explicit process of the reasoning step to obtain, in this explicit chain of thought reasoning, it will include the structural enhancement information obtained through which keywords.

[0080] At this time, through the large language model, the specified keywords are classified to determine the core keywords and flexible keywords contained therein. The natural query language and all the identified keywords are input, and the large language model classifies each keyword to identify the core keywords and flexible keywords among them. Among them, the flexible keywords refer to the keywords other than the core keywords.

[0081] At this time, the structural enhancement information corresponding to the core keywords is retained, and the structural enhancement information corresponding to at least some of the flexible keywords is restored to the corresponding flexible keywords to obtain the updated enhanced query language. That is, during the task execution of the reinforcement learning model, only the core keywords and at most some of the flexible keywords are converted into structural enhancement information, while the "at least some flexible keywords" mentioned above are retained as keywords without further conversion of structural enhancement information. At this time, the "at least some flexible keywords" can be reversely restored through the reinforcement learning model, or when performing the forward conversion, the "at least some flexible keywords" can be ignored and the conversion task can be re-executed.

[0082] After obtaining the updated enhanced query language, it can be used as an output of the current natural query language to determine the performance of this output according to the reward function and to guide the parameters of the reinforcement learning model.

[0083] Furthermore, when restoring or re-converting at least some of the flexible keywords, among all the flexible keywords, according to the selection quantity corresponding to the current training batch, the corresponding quantity of flexible keywords is selected as a keyword combination, and the structural enhancement information corresponding to the keyword combination is restored to the corresponding flexible keywords to obtain the updated enhanced query language and obtain its corresponding performance.

[0084] Among them, the training batch means that the quantity of flexible keywords selected for restoration is different in each training batch. For example, currently there are a total of 5 flexible keywords. In the first training batch, 4 flexible keywords are selected for restoration (at this time, there are a total of 4 selection methods). In the second training batch, 3 flexible keywords are selected for restoration (at this time, there are a total of 10 selection methods). In the third training batch, 2 flexible keywords are selected for restoration (at this time, there are a total of 10 selection methods).

[0085] Here, each selection method is regarded as a keyword combination in this training batch, from which the updated enhanced query language corresponding to each keyword combination is obtained, and the corresponding performance is obtained.

[0086] For all keyword combinations in each training batch, the updated enhanced query languages corresponding to several keyword combinations with the best performance in the specified dimension index among them are used to calculate the corresponding reward function and used for training the reinforcement learning model; the training progress of the reinforcement learning model is positively correlated with the number of selected keyword combinations.

[0087] Here, in order to concentrate resources on optimizing high-return strategies and improve training efficiency, the reward function is not calculated for each selection method (that is, each keyword combination). Only several keyword combinations with the best performance in the specified dimension index (such as recall rate) in each training batch are selected to calculate the reward function of their corresponding enhanced query languages and used for training the reinforcement learning model. This can guide the reinforcement learning model to selectively convert the keywords in the face of complex natural query languages, thereby providing a certain basic guarantee for the recall rate.

[0088] The earlier the training progress of the reinforcement learning model, that is, when the model training just starts, at this time, the choices in most cases should be considered, so the number of selected keyword combinations is small. For example, only one keyword combination with the best performance is selected in each batch. As the training progress of the reinforcement learning model lasts longer and is closer to the end of training, at this time, the model should be trained specifically to be able to handle special cases of complex natural query languages. Therefore, more keyword combinations can be selected to calculate the reward function for model training.

[0089] Such as Figure 3 shown, the embodiment of the present application also provides an NL2SQL model query enhancement device based on reinforcement learning, including: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the reinforcement learning-based NL2SQL model query enhancement method as described in any of the above embodiments.

[0090] The embodiment of the present application also provides a non-volatile computer storage medium storing computer-executable instructions, and the computer-executable instructions are set as: the reinforcement learning-based NL2SQL model query enhancement method as described in any of the above embodiments.

[0091] Each embodiment in this application is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other, and the differences between each embodiment and other embodiments are emphasized. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for relevant content.

[0092] The devices and media provided in the embodiments of this application correspond one-to-one with the methods. Therefore, the devices and media also have beneficial technical effects similar to those of their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be elaborated here.

[0093] The above description is only for the embodiments of this application and is not intended to limit this application. For those skilled in the art, various modifications and changes can be made to this application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.< / think> < / answer> < / think>

Claims

1. A query enhancement method for an NL2SQL model based on reinforcement learning, characterized in that, Including: Determine a reinforcement learning model for performing a query enhancement task, where the query enhancement task is used to output a corresponding enhanced query language according to a natural query language input by a user; Obtain multiple existing natural query languages, convert them into SQL query statements through an NL2SQL model and execute them for the natural query languages, determine context information corresponding to the structural information according to the execution results, and the performance of the context information in a preset multi-dimensional metric, so as to generate training samples; Based on the group relative policy optimization algorithm, train the reinforcement learning model through the training samples according to a preset reward function; the reward function is set based on the performance of the context information in the multi-dimensional metric corresponding to the natural query language and the enhanced query language respectively.

2. The query enhancement method of the NL2SQL model based on reinforcement learning according to claim 1, wherein Before obtaining the training samples generated according to the natural query language, the method further includes: Define context information according to the structural information of the database and integrate it to obtain a corresponding context set; Define the query enhancement task of the reinforcement learning model, set the natural query language as the state and the enhanced query language as the action; Establish an end-to-end training framework; the end-to-end training framework includes a group relative policy optimization algorithm and a reward function, the group relative policy optimization algorithm generates inference steps in the inference stage and generates an enhanced query language in the output stage; the end-to-end training framework performs iterative training based on the reward function; Establish query scenarios; there are multiple query scenarios, and the structural information of the databases in different query scenarios is different, which is used to perform a generality test on the reinforcement learning model.

3. The query enhancement method for the NL2SQL model based on reinforcement learning according to claim 1, characterized in that, The reward function includes a general part and a separate part, the general part includes multiple first metrics for evaluating the performance of the natural query language and the enhanced query language in the first metric; The separate part includes multiple second metrics for evaluating the performance of the enhanced query language in the second metric; The first metric includes accuracy, recall, and F1 metric, and the second metric includes query time and whether there is an error; The second metric is set with a corresponding weight factor.

4. The query enhancement method for the NL2SQL model based on reinforcement learning according to claim 3, characterized in that, Based on the group relative policy optimization algorithm, train the reinforcement learning model through the training samples according to a preset reward function, specifically including: Based on the group relative policy optimization algorithm, determine the corresponding reward function; Based on the general part of the reward function, select at least some of the first metrics as the current metrics, and based on the current metrics, train the reinforcement learning model through the training samples until the training expectation is met; the training expectation includes that the number of training rounds reaches a preset number of rounds, and / or the performance of the enhanced query language output by the reinforcement learning model reaches a preset standard; Among the remaining first metrics, continue to select at least some of the first metrics and add them to the current metrics, and train the reinforcement learning model based on the added current metrics until all the first metrics are added to the current metrics and the training expectation is met; Add the second metric to the current metric, set the weight factor corresponding to the second metric based on the training process of the first metric, and train the reinforcement learning model according to the current metric.

5. The method for query enhancement of the NL2SQL model based on reinforcement learning according to claim 4, wherein Setting the weight factor corresponding to the second metric based on the training process of the first metric specifically includes: For each round of training process of the first metric, perform weighted summation according to its corresponding single-round training efficiency and corresponding single-round performance to obtain the corresponding overall training efficiency and overall performance; wherein, when the current metric changes, enter the next round of training process, and the weight of the later training process is higher; Determine the weight factor corresponding to the second metric according to the overall training efficiency and the overall performance; wherein, the higher the overall training efficiency and the better the overall performance, the higher the weight factor corresponding to the second metric.

6. The method for query enhancement of the NL2SQL model based on reinforcement learning according to claim 1, wherein The method further includes: During the training process of the reinforcement learning model, if it is determined that the length of the structural enhancement information in the enhanced query language is higher than the preset length, and the performance of the specified dimension metric in the corresponding multi-dimensional metrics of the execution result obtained according to the enhanced query language is lower than the preset standard, then determine the specified keyword corresponding to it in the natural query language according to the structural enhancement information in the enhanced query language; Perform hierarchical processing on the specified keyword through a large language model to determine the core keyword and flexible keyword included therein; Retain the structural enhancement information corresponding to the core keyword, and restore the structural enhancement information corresponding to at least part of the flexible keyword to the corresponding flexible keyword to obtain the updated enhanced query language.

7. The query enhancement method for the NL2SQL model based on reinforcement learning according to claim 6, characterized in that, Restoring the structural enhancement information corresponding to at least part of the flexible keyword to the corresponding flexible keyword to obtain the updated enhanced query language specifically includes: Among all the flexible keywords, select the corresponding number of flexible keywords as the keyword combination according to the selection quantity corresponding to the current training batch, and restore the structural enhancement information corresponding to the keyword combination to the corresponding flexible keyword to obtain the updated enhanced query language, and obtain its corresponding performance; For all keyword combinations in each training batch, use the updated enhanced query languages corresponding to several keyword combinations with the best performance of the specified dimension metric among them to calculate the corresponding reward function and use it for training the reinforcement learning model; the training progress of the reinforcement learning model is positively correlated with the number of selected keyword combinations.

8. The method for query enhancement of the NL2SQL model based on reinforcement learning according to claim 2, wherein Establishing a query scenario specifically includes: During the training process of the reinforcement learning model, determine several most common context information corresponding to each execution result in the current query scenario; Generate alternative structure information for substitution according to the several most common context information to generate a new query scenario for training the reinforcement learning model.

9. An NL2SQL model query enhancement device based on reinforcement learning, characterized in that, Including: At least one processor; And, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for query enhancement of the NL2SQL model based on reinforcement learning as described in any one of claims 1 to 8.

10. A non-volatile computer storage medium stores computer-executable instructions, characterized in that, The computer-executable instructions are set to the method for query enhancement of the NL2SQL model based on reinforcement learning as described in any one of claims 1 to 8.

Images