Model training sample construction method and apparatus, device, storage medium, and computer program product

Abstract

The application relates to a model training sample construction method and device, equipment, a storage medium and a computer program product. The method comprises the following steps: performing node reasoning on a query question sample through an initial inference model obtained by pre-training to obtain a plurality of leaf nodes; performing node expansion on at least one leaf node to obtain a plurality of expansion nodes, and performing inference simulation on the plurality of expansion nodes until a terminal node is obtained; based on the terminal node, backtracking to a root node matched with the query question sample in sequence to obtain a plurality of reasoning thought chains; and constructing a training sample according to the query question sample and the plurality of reasoning thought chains, wherein the training sample is used for model fine-tuning of the initial inference model to obtain a target inference model; and the target inference model is used for question reasoning based on a query question to obtain a corresponding query plan. The method can improve the accuracy of reasoning query.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and computer program product for constructing training samples for a model. Background Technology

[0002] In the field of query optimization, the traditional cost-based optimizer (CBO) has long been a core component of databases. However, inaccurate statistical information can directly lead to biased cost estimation, resulting in the selection of suboptimal execution plans. Furthermore, when faced with complex queries, its enumeration space can expand exponentially, making it unable to efficiently traverse all possible plans or cope with flexible and ever-changing query patterns and data scenarios. These problems have long constrained the query execution efficiency of databases.

[0003] Currently, with the rapid development of large language models, optimization problems can be defined through offline model fine-tuning and prompt word optimization, and high-quality query optimization capabilities can be continuously learned online through retrieval enhancement generation technology. However, cloud database providers need to handle slow query issues from different clusters, instances, and users for extended periods. Large language models still heavily rely on statistical information or execution plans recommended by the original database optimizer, limiting the capabilities of the large model to the optimizer's capabilities. Consequently, the query plans output by large models still suffer from low accuracy. Therefore, improving the accuracy of inferential queries remains a pressing issue that needs to be addressed. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, device, storage medium, and computer program product for constructing training samples of a model to improve the accuracy of inference queries, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides a method for constructing training samples for a model. The method includes:

[0006] Using the initial inference model obtained through pre-training, multiple leaf nodes are obtained by performing node inference on the query question sample;

[0007] For at least one leaf node, expand the node to obtain multiple extended nodes, and perform inference simulation on the multiple extended nodes until the terminal node is obtained;

[0008] The reasoning chain is obtained by backtracking from the termination node to the root node that matches the query question sample, and multiple reasoning chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0009] Training samples are constructed based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0010] In one embodiment, performing inference simulation on multiple extended nodes until a termination node is obtained includes: performing inference simulation on multiple extended nodes to obtain multiple inference nodes; stopping the inference simulation when the inference simulation behavior performed by the inference node is a preset behavior, and selecting the inference node whose inference simulation behavior is the preset behavior among the multiple inference nodes as the termination node.

[0011] Secondly, this application provides a method for training a reasoning model. The method includes:

[0012] Obtain the training samples constructed using the training sample construction method of the model provided in the first aspect;

[0013] A predicted query plan is obtained based on the training samples and an initial inference model;

[0014] The first model loss is constructed based on the predicted query plan and training samples, and the initial inference model is fine-tuned using the first model loss to obtain the target inference model. The target inference model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0015] Thirdly, this application provides a method for problem reasoning. The method includes:

[0016] Obtain the query question, and perform node reasoning on the query question based on the target reasoning model obtained by the training sample construction method of the model provided in the first aspect to obtain the leaf node;

[0017] Expand the leaf nodes to obtain extended nodes, and perform reasoning simulation on the extended nodes until the terminal node is obtained;

[0018] Based on the termination node, we backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

[0019] Fourthly, this application also provides a training sample construction apparatus for a model. The apparatus includes:

[0020] The node reasoning module is used to perform node reasoning on the query question sample to obtain multiple leaf nodes using the initial reasoning model obtained through pre-training.

[0021] The node expansion and simulation module is used to expand at least one leaf node to obtain multiple expanded nodes, and to perform inference simulation on the multiple expanded nodes until the terminal node is obtained.

[0022] The node backtracking module is used to backtrack from the termination node to the root node that matches the query question sample to obtain the reasoning thought chain, and to obtain multiple reasoning thought chains by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0023] The training sample construction module is used to construct training samples based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0024] Fifthly, this application also provides a training device for a reasoning model. The device includes:

[0025] The training sample acquisition module is used to acquire training samples constructed using the training sample construction method of the model provided in the first aspect.

[0026] The inference prediction module is used to obtain a predicted query plan based on the training samples and an initial inference model.

[0027] The model training module is used to construct the first model loss according to the predicted query plan and training samples, and to fine-tune the initial inference model using the first model loss to obtain the target inference model; the target inference model is used to perform question inference based on the query question to obtain the corresponding query plan.

[0028] Sixthly, this application also provides a problem reasoning apparatus. The apparatus includes:

[0029] The node reasoning module is used to obtain the query question and perform node reasoning on the query question based on the target reasoning model obtained by the training method of the reasoning model provided in the second aspect to obtain the leaf nodes.

[0030] The node expansion module is used to expand leaf nodes to obtain expanded nodes, and to perform inference simulation on the expanded nodes until the terminal node is obtained.

[0031] The question reasoning module is used to backtrack sequentially from the termination node to the root node that matches the query question, and obtain the query plan corresponding to the query question.

[0032] Seventhly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0033] Using the initial inference model obtained through pre-training, multiple leaf nodes are obtained by performing node inference on the query question sample;

[0034] For at least one leaf node, expand the node to obtain multiple extended nodes, and perform inference simulation on the multiple extended nodes until the terminal node is obtained;

[0035] The reasoning chain is obtained by backtracking from the termination node to the root node that matches the query question sample, and multiple reasoning chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0036] Training samples are constructed based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0037] Eighthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0038] Obtain the training samples constructed using the training sample construction method of the model provided in the first aspect;

[0039] A predicted query plan is obtained based on the training samples and an initial inference model;

[0040] The first model loss is constructed based on the predicted query plan and training samples, and the initial inference model is fine-tuned using the first model loss to obtain the target inference model. The target inference model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0041] Ninthly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0042] Obtain the query question, and perform node reasoning on the query question based on the target reasoning model obtained by the training method of the reasoning model provided in the second aspect to obtain the leaf node;

[0043] Expand the leaf nodes to obtain extended nodes, and perform reasoning simulation on the extended nodes until the terminal node is obtained;

[0044] Based on the termination node, we backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

[0045] Tenthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0046] Using the initial inference model obtained through pre-training, multiple leaf nodes are obtained by performing node inference on the query question sample;

[0047] For at least one leaf node, expand the node to obtain multiple extended nodes, and perform inference simulation on the multiple extended nodes until the terminal node is obtained;

[0048] The reasoning chain is obtained by backtracking from the termination node to the root node that matches the query question sample, and multiple reasoning chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0049] Training samples are constructed based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0050] Eleventhly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0051] Obtain the training samples constructed using the training sample construction method of the model provided in the first aspect;

[0052] A predicted query plan is obtained based on the training samples and an initial inference model;

[0053] The first model loss is constructed based on the predicted query plan and training samples, and the initial inference model is fine-tuned using the first model loss to obtain the target inference model. The target inference model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0054] In a twelfth aspect, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0055] Obtain the query question, and perform node reasoning on the query question based on the target reasoning model obtained by the training method of the reasoning model provided in the second aspect to obtain the leaf node;

[0056] Expand the leaf nodes to obtain extended nodes, and perform reasoning simulation on the extended nodes until the terminal node is obtained;

[0057] Based on the termination node, we backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

[0058] In a thirteenth aspect, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0059] Using the initial inference model obtained through pre-training, multiple leaf nodes are obtained by performing node inference on the query question sample;

[0060] For at least one leaf node, expand the node to obtain multiple extended nodes, and perform inference simulation on the multiple extended nodes until the terminal node is obtained;

[0061] The reasoning chain is obtained by backtracking from the termination node to the root node that matches the query question sample, and multiple reasoning chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0062] Training samples are constructed based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0063] In a fourteenth aspect, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0064] Obtain the training samples constructed using the training sample construction method of the model provided in the first aspect;

[0065] A predicted query plan is obtained based on the training samples and an initial inference model;

[0066] The first model loss is constructed based on the predicted query plan and training samples, and the initial inference model is fine-tuned using the first model loss to obtain the target inference model. The target inference model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0067] In a fifteenth aspect, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0068] Obtain the query question, and perform node reasoning on the query question based on the target reasoning model obtained by the training method of the reasoning model provided in the second aspect to obtain the leaf node;

[0069] Expand the leaf nodes to obtain extended nodes, and perform reasoning simulation on the extended nodes until the terminal node is obtained;

[0070] Based on the termination node, we backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

[0071] The training sample construction method, apparatus, computer equipment, storage medium, and computer program product of the above model, through the initial inference model obtained by pre-training, perform node inference on the query question sample to obtain multiple leaf nodes; expand at least one leaf node to obtain multiple extended nodes, and perform inference simulation on multiple extended nodes until a terminal node is obtained; backtrack to the root node matching the query question sample based on the terminal node to obtain an inference thought chain, and obtain multiple inference thought chains by repeatedly executing node inference, node expansion, inference simulation, and backtracking; construct training samples based on the query question sample and multiple inference thought chains, wherein the training samples are used to fine-tune the initial inference model to obtain the target inference model; the target inference model is used to perform question inference based on the query question to obtain the corresponding query plan. By performing node reasoning on query question samples, leaf nodes that meet the query question requirements can be selected progressively from the root node. Node expansion is then performed on these leaf nodes to ensure the completeness of node exploration. By backtracking from the terminating node to the root node, multiple reasoning chains are obtained. This allows for the gradual focus on more optimized reasoning chains through high-value nodes. Training samples are then constructed using these reasoning chains and query question samples to fine-tune the initial reasoning model. This enables the initial reasoning model to learn from the optimized reasoning chains how to more accurately and efficiently obtain reasoning nodes and paths. The resulting target reasoning model outputs a query plan that better meets the query question's requirements, thereby improving query performance and ultimately enhancing the accuracy of reasoning queries based on the target reasoning model. Attached Figure Description

[0072] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0073] Figure 1 This is a diagram illustrating the application environment of a training sample construction method for a model in one embodiment.

[0074] Figure 2 This is a flowchart illustrating the process of obtaining the target inference model and the target process reward model in an offline phase, as shown in one embodiment.

[0075] Figure 3 This is a flowchart illustrating the method for constructing training samples for a model in one embodiment.

[0076] Figure 4 This is a schematic diagram illustrating the selection of the optimal child node from multiple candidate child nodes in one embodiment.

[0077] Figure 5 This is a schematic diagram illustrating how leaf nodes are obtained by continuing the search downwards along the tree structure using the optimal child node in one embodiment.

[0078] Figure 6 This is a schematic diagram illustrating multiple reasoning actions in one embodiment;

[0079] Figure 7 This is a schematic diagram illustrating multiple candidate child nodes obtained in one embodiment;

[0080] Figure 8 This is a schematic diagram illustrating the reasoning simulation based on the target expansion node until the termination node is obtained in one embodiment;

[0081] Figure 9 This is a schematic diagram of the reasoning thought chain obtained in one embodiment;

[0082] Figure 10 This is a schematic diagram illustrating how multiple reasoning thought chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation, and backtracking in one embodiment.

[0083] Figure 11 This is a flowchart illustrating the method for constructing training samples for a model in another embodiment;

[0084] Figure 12 This is a flowchart illustrating the training method of an inference model in one embodiment;

[0085] Figure 13 This is a flowchart illustrating a problem reasoning method in one embodiment;

[0086] Figure 14 This is a schematic diagram illustrating node expansion during the problem reasoning process in one embodiment;

[0087] Figure 15 This is a flowchart illustrating the process of node selection, node expansion, reasoning simulation, and backtracking during problem reasoning in one embodiment.

[0088] Figure 16 This is a complete flowchart illustrating the model's training sample construction method, inference model training method, and problem inference method in one embodiment.

[0089] Figure 17This is a structural block diagram of a training sample construction device for a model in one embodiment;

[0090] Figure 18 This is a structural block diagram of a training device for an inference model in one embodiment;

[0091] Figure 19 This is a structural block diagram of the problem reasoning device in one embodiment;

[0092] Figure 20 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0093] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0094] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0095] In the field of query optimization, traditional cost-based optimizers have long been a core component of databases. However, inaccurate statistical information (such as skewed data distribution and outdated statistics due to dynamic data updates) directly leads to biased cost estimation, resulting in the selection of suboptimal execution plans. Furthermore, when facing complex queries (such as multi-table joins and nested subqueries), their enumeration space expands exponentially, making it impossible to efficiently traverse all possible plans or handle flexible and ever-changing query patterns and data scenarios. These problems have long constrained the query execution efficiency of databases. Currently, leveraging artificial intelligence to enhance query optimizers has become a mainstream research trend. Small-model-based methods extract information such as tables, operators, and estimated costs from each node of a tree-structured execution plan as features and train a tree-structured convolutional neural network to capture the comparison between good and bad execution plans. With the rapid development of large language models, optimization problems can be defined through offline model fine-tuning and prompt word optimization, and high-quality query optimization capabilities can be continuously learned online through retrieval enhancement generation techniques.

[0096] However, the training of small models heavily relies on local data and execution scenarios of a single instance. Their training data depends on the actual query execution of the instance or standby machine. The constraint of "not occupying tenant resources" in cloud scenarios directly cuts off the effective training data source for small models, fundamentally limiting their generalization ability. Furthermore, cloud database providers need to handle slow query issues from different clusters, instances, and users in the long term. Large language models still heavily rely on the statistical information provided by the original database optimizer or the execution plan recommended by the optimizer, thus limiting the capabilities of large models to the optimizer's capabilities. Therefore, the query plans output by large models still suffer from low accuracy. Thus, improving the accuracy of inference queries remains a pressing issue that needs to be addressed.

[0097] This application provides a method for constructing training samples for a model to improve the accuracy of inference queries, a method for training an inference model, and a method for question reasoning. The method for constructing training samples, training an inference model, and providing question reasoning provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another server.

[0098] Specifically, taking server 104 as an example, the method for constructing training samples for the model is explained. Server 104 uses the initial inference model obtained through pre-training to perform node inference on the query question sample to obtain multiple leaf nodes, and performs node expansion on at least one leaf node to obtain multiple extended nodes. Inference simulation is then performed on the multiple extended nodes until a termination node is obtained. Based on the termination node, the model backtracks to the root node that matches the query question sample to obtain an inference thought chain. Multiple inference thought chains are obtained by repeatedly executing node inference, node expansion, inference simulation, and backtracking. Training samples are then constructed based on the query question sample and multiple inference thought chains. The aforementioned training samples are used to fine-tune the initial inference model to obtain the target inference model. The obtained target inference model is used to perform question inference based on the query question to obtain the corresponding query plan. By expanding leaf nodes to ensure the completeness of node exploration and gradually focusing on more optimized reasoning chains through high-value nodes, the initial reasoning model is fine-tuned by constructing training samples using reasoning chains and query question samples. This allows the initial reasoning model to learn from more optimized reasoning chains how to obtain reasoning nodes and reasoning paths more accurately and efficiently. As a result, the query plan output by the trained target reasoning model better meets the requirements of the query question, thereby improving the model's query performance and thus enhancing the accuracy of reasoning queries based on the target reasoning model.

[0099] Secondly, taking server 104 as an example, the training method of the inference model is explained. Server 104 obtains the training samples constructed by the aforementioned model training sample construction method, and then obtains the predicted query plan based on the training samples through the initial inference model. Finally, a first model loss is constructed according to the predicted query plan and the training samples, and the initial inference model is fine-tuned using the first model loss to obtain the target inference model. The aforementioned target inference model is used to perform question reasoning based on the query question to obtain the corresponding query plan. Since the initial inference model is fine-tuned by constructing training samples using the reasoning thought chain and query question samples, the initial inference model can learn from a more optimized reasoning thought chain how to obtain reasoning nodes and reasoning paths more accurately and efficiently. Therefore, the query plan output by the target inference model obtained through training is more in line with the requirements of the query question, thereby improving the query performance of the model and thus improving the accuracy of inference queries based on the target inference model.

[0100] Finally, taking server 104 as an example, the method of question reasoning is explained. Server 104 obtains the query question and, through the target reasoning model obtained by the training sample construction method of the aforementioned model, performs node reasoning on the query question to obtain leaf nodes. Then, it expands the leaf nodes to obtain extended nodes, and performs reasoning simulation on the extended nodes until a termination node is obtained. Based on the termination node, it backtracks sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question. Since the query plan output by the target reasoning model obtained through training is more in line with the requirements of the query question, the accuracy of reasoning queries based on the target reasoning model is improved.

[0101] The terminal 102 can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The training sample construction method for the model provided in this application embodiment can be applied to various scenarios, including but not limited to scenarios such as problem reasoning in a database and querying in a cloud database.

[0102] The following will introduce the relevant technical concepts involved in the training sample construction method, the inference model training method, and the problem reasoning method of the embodiments of this application:

[0103] Query optimization (QO) is a core function of relational database management systems. It aims to generate optimal execution plans for Structured Query Language (SQL) queries to minimize CPU, memory, I / O resource consumption and improve query response speed. QO is primarily achieved through logical-level optimization (such as equivalent rewriting of relational algebra expressions and pre-executing selection and projection operations to reduce data size) and physical-level optimization (such as selecting the optimal access path and join algorithm based on data statistics and index information), ultimately outputting a directly executable physical query plan. Generally, in this embodiment, query optimization refers to cost-based query optimization (CQO).

[0104] Database hints (Hints): This refers to a specific annotation syntax in the database field. Users or developers actively inject these hints into SQL statements (or database operation instructions) to pass additional execution strategy information (such as index selection, join methods, execution order, etc.) to the database query optimizer. This helps or constrains the optimizer to generate an execution plan that better meets actual needs, rather than simply "guiding and prompting the database to perform corresponding operations".

[0105] Large Language Model (LLM): A large language model is an artificial intelligence model built on deep learning technology. Its core is a deep artificial neural network containing billions to hundreds of billions of parameters. Large language models are pre-trained on large-scale, diverse unlabeled text data through self-supervised learning (primarily) or semi-supervised learning paradigms. The goal is to learn the syntax, semantics, logic, and world knowledge of human language, thereby achieving the ability to generate high-quality natural language, understand textual intent, and complete language-related tasks (such as question answering, summarizing, and translation).

[0106] Monte Carlo Tree Search (MCTS) is a reinforcement learning algorithm suitable for complex decision-making processes. Its core logic combines the generality of stochastic simulation with the precision of tree-based search, iteratively optimizing the inference tree through a four-step process: selection, expansion, simulation, and backtracking. Specifically, it selects the optimal child node from the root node down to the leaf node, expands the child nodes of incompletely explored leaf nodes, simulates the subsequent process using a stochastic strategy from the newly expanded nodes, and then updates all nodes on the path from the root node with the simulation results. This process gradually focuses on high-value decision paths and is widely used in game theory (such as Go and chess), path planning, combinatorial optimization, and other scenarios requiring sequential decision-making.

[0107] Action Value (Q-Value): The Q-value is a core metric in reinforcement learning used to quantify the expected cumulative reward of an agent after performing an action in a specific state. The core function of the Q-value is to evaluate the long-term value of the decision to "choose action a in the current state s," that is, the expected total reward the agent can obtain by following the optimal policy after performing the action. It directly guides the agent's action selection strategy (e.g., prioritizing actions with higher Q-values).

[0108] Rollout: Rollout is a core operation in reinforcement learning (especially in simulation-based algorithms like MCTS). Essentially, it starts from an intermediate state in the decision-making process (usually an incompletely explored node) and, using a pre-defined, relatively simple "rollout policy," rapidly simulates a series of subsequent actions until the task's termination state is reached, thereby generating a complete decision sequence and obtaining the final reward. The core purpose of this process is to quickly and stochastically estimate the long-term potential value of the current node without complex computation or optimal selection, providing an evaluation basis for higher-level decision-making algorithms (such as the selection and backtracking stages of MCTS). It plays the role of "rapid look-ahead and value estimation" and is a key technique for balancing search depth and breadth.

[0109] Supervised Fine-Tuning (SFT): This involves adapting the general knowledge learned during the pre-training phase to specific downstream tasks. Specifically, it involves constructing labeled datasets containing "input-expected output" (such as "question-standard answer" for question-answering tasks and "source text-target text" for translation tasks) and adjusting model parameters in a supervised learning manner. This makes the model's prediction results on the target task more closely match the correct direction of human annotation, ultimately improving the model's task performance and output accuracy in specific scenarios. It is the core step for large language models to shift from "general capabilities" to "specific capabilities".

[0110] Process Reward Model (PRM): A process reward model is a model used in reinforcement learning to provide dynamic reward feedback for the decision-making process of an agent. Its core function is to break the limitation of the traditional "terminal reward" which only provides single feedback at the end of the task. For each step or stage behavior of the agent from the initial state to the target state, the PRM calculates and outputs the reward value in real time according to the preset evaluation criteria (such as the rationality of the action, the closeness to the target, the efficiency of the steps, etc.), so as to more accurately guide the agent to learn the optimal intermediate decision-making strategy. It is commonly used in reinforcement learning training of complex long sequence tasks (such as path planning and multi-step reasoning).

[0111] The training sample construction method provided in this embodiment is used to construct training samples for training the inference model. This allows the inference model training method to obtain a target inference model and a target process reward model through the constructed training samples. This enables the problem inference method to obtain a query plan corresponding to the query question through the target inference model and the target process reward model. Therefore, the goal of the query optimizer expert inference model in this embodiment is to: input a given query, default plan execution information, and relevant auxiliary information, so as to output an accurate thought chain and recommended intervention configuration (i.e., the final optimizer hint set "Hint Set" obtained through inference). That is, the query optimizer expert inference model obtained in this embodiment consists of a target inference model and a target process reward model. In practical applications, problem inference using the target inference model and the target process reward model is divided into two parts: an online stage and an offline stage, which will be described in detail below:

[0112] In the online phase: First, it's important to clarify that the query optimizer expert reasoning model will ultimately be integrated into SQL Plan Management (SPM). SPM has accepted plans (plan baselines) and plans to be validated (Plan History) for each query template. In this embodiment, only one plan is selected from the accepted plans for execution. The SPM activation mechanism is as follows: When processing a specific query, a plan is first obtained through the traditional query optimizer. This plan is then checked against the plan baseline. If it exists, it is directly selected for execution; otherwise, the lowest-cost plan is selected from the plan baseline. Finally, it is placed in the plan pool to be validated. Therefore, in this embodiment, the target reasoning model and the target process reward model trigger new plans to enter the Plan History. For detected slow queries, the general query optimizer expert reasoning model is asynchronously invoked, inputting the current query, default execution information, and auxiliary information to generate a thought process and an intervention set (Hint set). This intervention set corresponds to an execution plan and is added to SPM as a plan to be validated. It also performs automatic plan evolution, which involves small-scale verification of the performance of plans in the Plan History, and only those that outperform the plan baseline are accepted.

[0113] Offline Phase: To obtain the target inference model and the target process reward model, it is necessary to fine-tune the general language model, which involves generating thought chains, training the inference model, and training the process reward model using the methods provided in the embodiments of this application. Specifically, as follows... Figure 2 The diagram shows the process of obtaining the target inference model and the target process reward model in the offline stage. The training of the inference model and the process reward model are iterative and progressive. For the query question sample, the initial inference model is used to think step by step, and the inference thought chain is explored based on the action space designed in Monte Carlo tree search. At this time, the initial process reward model can be used to initialize the scoring of the next multiple actions in combination with the current state. In this process, Monte Carlo tree search will automatically record the Q value of each action, thereby obtaining multiple inference thought chains with score labels.

[0114] At this point, for each query question sample, the reasoning thought chain with the highest average Q value (i.e., the positive sample thought chain) is selected as training data for fine-tuning the initial recommendation model. This means that training samples are constructed using the positive sample thought chain and the query question sample. Based on these training samples, the initial reasoning model is fine-tuned to obtain the target reasoning model, which then performs question reasoning based on the query question to derive the corresponding query plan. Similarly, for each query question sample, the reasoning thought chain with the highest average Q value (i.e., the positive sample thought chain) and the reasoning thought chain with the lowest average Q value (i.e., the negative sample thought chain) are selected as training data for fine-tuning the initial process reward model. This means that training samples are constructed using the positive sample thought chain and the query question sample, and vice versa. Based on these training samples, the initial process reward model is fine-tuned to obtain the target process reward model, which assigns a cumulative exploration value to the expanded nodes during the node expansion process based on the query question.

[0115] The above describes methods for obtaining the target inference model and the target process reward model in the offline stage. The training data foundation for obtaining these models is the construction of training samples. Therefore, the following specific examples illustrate how to construct training samples for the model: In an exemplary embodiment, as shown... Figure 3 As shown, a method for constructing training samples for a model is provided, which can be applied to... Figure 1 Taking server 104 as an example, it can be understood that this method can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. In this embodiment, the method includes steps 302 to 308. Wherein:

[0116] Step 302: Using the initial inference model obtained from pre-training, perform node inference on the query question sample to obtain multiple leaf nodes.

[0117] The initial inference model is a policy model used to derive a query plan based on the query question. In other words, the initial inference model is a pre-trained large model capable of providing the next intermediate inference state or the final answer based on the question, the current inference state, and the guiding direction. Secondly, the query question sample includes SQL query statements from the database management system and auxiliary information, including but not limited to database schema information, statistics, execution plans, system configuration, and environment information. Therefore, the initial inference model can predict the next character (Token) for the query question sample through autoregression, thereby generating a series of intermediate inference states leading to the final inference result for the query question sample. That is, the intermediate inference state generated by the initial inference model in step i is The aforementioned intermediate reasoning state This can be represented as a reasoning node in a reasoning tree, and the node state of the reasoning node is also the state included in the intermediate reasoning states. It also includes the currently recommended set of query intervention hints (Hint Set), which specifically includes... Therefore, the leaf nodes are the incompletely explored child nodes generated by the initial inference model in step i.

[0118] Specifically, the server first obtains a query question sample. This sample can be a historical SQL query stored on the server or an SQL query obtained through data collection; there are no specific limitations here. Based on this, the server calls an initial inference model obtained through pre-training with inference capabilities to infer the obtained query question sample. This yields multiple intermediate inference states leading to the final inference result for the query question sample. Each intermediate inference state can be represented as a inference node in the inference tree. Therefore, from the multiple inference nodes in the inference tree, leaf nodes that are not fully explored and are not terminal nodes are identified, thus completing node selection.

[0119] As described above, the method for selecting leaf nodes is as follows: a Monte Carlo tree search is used to search downwards through the currently maintained inference tree, selecting the optimal child node from the inference tree until a leaf node is found. The selection method for leaf nodes is described in detail below: In an exemplary embodiment, node inference is performed on the query question sample to obtain multiple leaf nodes, including: using the query question sample as the root node to search downwards for multiple candidate child nodes; selecting the optimal child node from the multiple candidate child nodes according to the upper confidence bound of each candidate child node; and continuing the search downwards along the tree structure using the optimal child node to obtain the leaf node.

[0120] The root node in the inference tree represents a sample query question containing the SQL query statement and auxiliary information. The upper confidence bound is the Upper Confidence Bound Apply to Tree (UCT). The upper bound of the confidence interval refers to the upper limit of the possible range of the estimated population parameter at a given confidence level. The upper bound of the confidence interval is usually calculated using the following formula: Confidence Interval = Sample Mean + (Critical Value × Standard Error). The standard error is the sample standard deviation divided by the square root of the sample size. The critical value depends on the chosen confidence level; for example, for a 95% confidence level, the critical value is typically 1.96 (under a normal distribution). Specifically, the query question sample is used as the root node in the inference tree to search downwards, resulting in multiple candidate child nodes for the root node. At this point, it is necessary to determine whether the candidate child nodes have been fully explored. If not, that is, the candidate child nodes have not been fully explored, new child nodes can be generated for the candidate child nodes. This determines the upper confidence bound of each candidate child node. Then, the candidate child node with the largest upper confidence bound value is selected from the multiple candidate child nodes as the optimal child node of the root node.

[0121] To facilitate understanding how to calculate the upper confidence bound of candidate child nodes, the calculation example is shown in the following formula (1):

[0122] (1)

[0123] in, This represents the upper confidence bound of candidate child node s. This indicates the number of times the candidate child node s has been visited. This indicates the number of times the parent node of the candidate child node s has been visited. This represents the explorable value of candidate child node s. Formula (1) allows the upper confidence bound of candidate child node s to achieve a balance between node exploration and node utilization.

[0124] For ease of understanding, such as Figure 4The diagram shown illustrates the selection of the optimal child node from multiple candidate child nodes. Taking the query question sample as the root node 401, the search proceeds downwards in the inference tree to obtain candidate child nodes 402, 403, and 404 for the root node 401. If the upper confidence bound of candidate child node 403 is the largest among the upper confidence bounds of candidate child nodes 402 to 404, then candidate child node 403 is selected as the optimal child node of the root node 401.

[0125] Understandably, determining whether a candidate child node has been fully explored requires at least one of the following methods: If the candidate child node is a fully expanded node, then all child nodes that the candidate child node can connect to have been generated and added to the tree, indicating that the candidate child node has been fully explored. Alternatively, all available actions of the candidate child node have been explored, indicating that the candidate child node has been fully explored. Or, no new child nodes can be generated for the candidate child node, indicating that the candidate child node has been fully explored. Therefore, if a candidate child node does not meet any of the above conditions, then the candidate child node has not been fully explored.

[0126] Furthermore, the server continues searching downwards along the tree structure using the optimal child node, following similar steps as described above, until it reaches an inference node that has not been fully explored or reaches a termination node. The incompletely explored inference node is designated as a leaf node. In other words, by continuing the search downwards along the tree structure using the optimal child node, multiple candidate child nodes for the optimal child node can be obtained. Based on the upper confidence bound of each candidate child node for the optimal child node, the candidate child node with the largest value is selected as the optimal child node of the optimal child node. If the optimal child node of the optimal child node has not been fully explored, this node can be designated as a leaf node. If the optimal child node of the optimal child node has been fully explored, this node becomes the termination node.

[0127] For ease of understanding, such as Figure 5As shown, by continuing the search downwards along the tree structure from the optimal child node, leaf nodes are obtained. Taking the query sample as the root node 501, the search downwards in the inference tree yields candidate child nodes 502, 503, and 504 for the root node 501. If the upper confidence bound of candidate child node 503 is the largest among the upper confidence bounds of candidate child nodes 502 to 504, then candidate child node 503 is selected as the optimal child node of the root node 501. Then, based on candidate child node 503 (i.e., the optimal child node of the root node 501), the search downwards in the inference tree yields candidate child nodes 505 and 506 for candidate child node 503. If the upper confidence bound of candidate child node 506 is the largest, then candidate child node 506 is selected as the optimal child node of candidate child node 503. If it can be determined that candidate child node 506 has not been fully explored, then candidate child node 506 is selected as a leaf node.

[0128] Understandably, if the termination node can be reached before node expansion, the inference thought chain described in step 306 can be directly obtained by backtracking from the termination node to the root node matching the query question sample. Multiple inference thought chains can then be obtained by repeatedly executing node inference, node expansion, inference simulation, and backtracking. The aforementioned inference nodes that have not been fully explored specifically satisfy any of the following conditions: the inference node still has ungenerated legitimate child nodes; or, the actionable actions within the inference node have not yet been expanded into child nodes; or, the inference node can continue to generate new child nodes.

[0129] Step 304: Expand at least one leaf node to obtain multiple extended nodes, and perform inference simulation on the multiple extended nodes until the termination node is obtained.

[0130] The termination node can be either a final termination node or a temporary termination node. A final termination node is the inference node where the executed inference simulation behavior is a preset behavior, while a temporary termination node is the inference node that generates a new execution plan. Specifically, after obtaining the leaf nodes, the server expands at least one leaf node to obtain multiple extended nodes for that leaf node. That is, for each leaf node, the server constructs a prompt word (propmt) based on the node state matched by the leaf node and different action guidance information, so that the initial inference model generates multiple candidate child nodes based on the node state and action guidance information included in the prompt word, and then selects an extended node from the multiple candidate child nodes. The node state matched by the aforementioned leaf node is the intermediate inference state described in the aforementioned embodiment. That is, if the leaf node is the action in step i, then the node state matched by the leaf node is the intermediate inference state generated in step i.

[0131] To facilitate understanding of the specific implementation of node expansion, the following describes how node expansion is performed: In an exemplary embodiment, node expansion is performed on at least one leaf node to obtain multiple expanded nodes, including: expanding the node based on the node state of the leaf node that satisfies the node expansion condition to obtain multiple candidate child nodes; and selecting an expanded node from the multiple candidate child nodes.

[0132] The node expansion condition indicates that a leaf node has not been fully explored; that is, a leaf node that meets the node expansion condition has not been fully explored. Determining whether a leaf node has been fully explored typically refers to whether all its child nodes have been generated and visited. Therefore, leaf nodes that can be expanded are those that have not been fully explored and are not final termination nodes. In other words, if a leaf node is a final termination node—meaning it has found an execution plan or reached a preset depth limit—then it does not need to be expanded further, and it will have no more child nodes, thus being considered fully explored. For leaf nodes that are not final termination nodes, if all possible child nodes are generated based on the leaf node, and each generated child node is visited (i.e., each generated child node is selected and simulated / backtracked), then the leaf node is also considered fully explored. Therefore, a leaf node not being fully explored includes at least: the existence of logical actions that have not yet generated child nodes, meaning the leaf node can still generate new child nodes; or, the unvisited candidate child nodes of the leaf node, meaning the generated candidate child nodes have not been selected for simulation or backtracking.

[0133] Specifically, after determining the leaf nodes through the aforementioned embodiments, the node state matched by each leaf node can also be obtained. At this point, prompt words are constructed according to the node states matched by the leaf nodes and different action guidance information, thereby enabling the initial inference model to generate multiple candidate child nodes based on the node states and action guidance information included in the prompt words. As described above, node expansion requires constructing prompt words using node states and different action guidance information. Different action guidance information indicates different inference actions. This application provides six different inference actions, specifically the first inference action, the second inference action, the third inference action, the fourth inference action, the fifth inference action, and the sixth inference action. For ease of understanding, as follows... Figure 6 The various reasoning actions shown.

[0134] The aforementioned first reasoning action is specifically single-step thinking - index selection, the second reasoning action is specifically single-step thinking - modifying the connection order, the third reasoning action is specifically single-step thinking - modifying the connection order, the fourth reasoning action is specifically raising a new sub-question, the fifth reasoning action is specifically rethinking, and the sixth reasoning action is specifically answering the question.

[0135] The system comprises four inference actions: The first inference action performs a slow-thinking function, meaning it iteratively reasons based on the current problem, modifies index selection, and generates a next step of reasoning based on the problem and existing steps. The possible inference actions following the first action are the first, fourth, fifth, and sixth inference actions. The second inference action performs reasoning based on the current problem, modifying the connection order. The possible inference actions following the second action are the second, fourth, fifth, and sixth inference actions. The third inference action investigates existing problems in the current configuration, performing reasoning such as setting `index_merge=on`, `hash_join=on`, `semijoin=on`, and `hypergraph_optimizer=off`. The possible inference actions following the third action are the second, fourth, fifth, and sixth inference actions.

[0136] Secondly, the fourth reasoning action is used to check whether there are still other problems in the current application's revised execution plan based on the input question and existing reasoning steps. If so, a new sub-question is proposed. The reasoning actions that can be executed after the fourth action are the first, second, third, and sixth reasoning actions. The fifth reasoning action, considering the possibility of errors in the reasoning process, is used to generate a new node based on the current node's parent node and the current node itself. That is, considering the result of the parent node, if the original node result is deemed incorrect, a new node is generated. Since the fifth reasoning action involves rethinking, the reasoning actions that can be executed after the fifth action are the first, second, third, fourth, fifth, and sixth reasoning actions. The sixth reasoning action indicates that there is no further room for optimization, stops the thinking process, and provides an answer. The reasoning node that executes the sixth reasoning action is the final termination node.

[0137] Therefore, given six reasoning actions, six prompt words can be constructed based on the node states matched by the leaf nodes and the action guidance information matched by the six reasoning actions. In this case, the initial reasoning model can generate a maximum of six candidate child nodes based on the prompt words. And at least one candidate child node can be obtained. For ease of understanding, the leaf node obtained in the aforementioned embodiment will be further described, such as... Figure 7 As shown, by expanding the node according to the node state matched by leaf node 701 and various action guidance information, candidate child nodes 702, 703, 704 and 705 can be obtained.

[0138] Furthermore, after expanding from leaf nodes to obtain candidate child nodes, it is necessary to select the expansion node capable of inference simulation from multiple candidate child nodes. This means determining whether the candidate child node is a new inference node or constitutes a new execution plan, avoiding repeated traversal and expansion of the same inference node within the tree structure. Different candidate child nodes may add new database hints or may not add new database hints, only exhibiting the inference process. Therefore, candidate child nodes that add database hints and those that do not are filtered separately to select the expansion node from multiple candidate child nodes. This will be explained in detail below:

[0139] First, we introduce how to filter candidate child nodes for new database prompt information. In an exemplary embodiment, selecting an extension node from multiple candidate child nodes includes: when a candidate child node contains new database prompt information, performing information matching between the new database prompt information and existing database prompt information to obtain an information matching result; if the information matching result indicates an information conflict, pruning the candidate child node; wherein, an information conflict exists because there is an information conflict between the new database prompt information and the existing database prompt information; selecting a candidate child node whose information matching result indicates no information conflict as an extension node; wherein, no information conflict exists because there is no information conflict between the new database prompt information and the existing database prompt information.

[0140] Specifically, when a candidate child node contains a hint about adding a new database, an Explain verification is performed on that node. This Explain verification verifies the execution plan analysis and conflict detection process performed on the candidate child node during the node expansion phase, thus validating the technical feasibility and effectiveness of adding the hint. Therefore, at least one of the following needs to be verified: whether the hint about adding a new database conflicts with existing hints; whether the hint about adding a new database actually changes the execution plan; or whether the hint about adding a new database is executable in the current database environment.

[0141] Based on this, to check whether there is a conflict between the new database prompt and the existing database prompt, information matching can be performed between the two prompts to obtain the matching result. The matching result indicates whether there is a conflict or not. A conflict exists between the new and existing database prompts, while no conflict exists. Therefore, if the matching result indicates a conflict, the Explain verification of the candidate child node fails. In this case, the candidate child node is pruned, and it will no longer be used as an expansion node. Furthermore, the pruned candidate child node will not be included in the inference tree, nor will it be included in the subsequent inference thought chain. Conversely, if the matching result indicates no conflict, the Explain verification of the candidate child node passes, and the candidate child node with no conflict is selected as the expansion node.

[0142] As described above, Explain verification can also check whether the new database prompt message has actually changed the execution plan, or verify whether the new database prompt message is executable in the current database environment. This is described below: In an exemplary embodiment, selecting an extension node from multiple candidate child nodes includes: if a candidate child node has a new database prompt message, determining the extended execution plan of the candidate child node and comparing it with the current execution plan of the leaf node that meets the node extension conditions; if the extended execution plan is consistent with the current execution plan, pruning the candidate child node; and selecting the candidate child node whose extended execution plan is inconsistent with the current execution plan as the extension node.

[0143] Specifically, when a candidate child node contains a "new database" hint, the `Explain` statement is used to verify whether this hint actually changes the execution plan. First, the extended execution plan of the candidate child node is determined, and then compared with the current execution plan of the leaf node that meets the node extension condition. Next, it is checked whether the extended execution plan and the current execution plan are consistent. If they are consistent, it means the "new database" hint has not actually changed the execution plan, and the candidate child node is still pruned. The pruned candidate child node will no longer be considered an extension node. If they are inconsistent, it means the "new database" hint has actually changed the execution plan, and the candidate child node with the inconsistent extended execution plan is selected as the extension node.

[0144] Similarly, considering that Explain can also verify whether the "Add Database" prompt is executable in the current database environment, if the "Add Database" prompt cannot be executed in the current database environment, the candidate child nodes will be pruned, and the pruned candidate child nodes will no longer be used as expansion nodes. Conversely, if the "Add Database" prompt is executable in the current database environment, the candidate child nodes will be used as expansion nodes.

[0145] The above describes how to filter candidate child nodes for new database prompt information. The following will describe in detail how to filter candidate child nodes for which no new database prompt information has been added. In an exemplary embodiment, selecting an extension node from multiple candidate child nodes includes: determining the cumulative exploration value of the candidate child node when no new database prompt information has been added; and selecting the candidate child node whose cumulative exploration value meets the filtering criteria from multiple candidate child nodes as the extension node.

[0146] Specifically, when no new database hints are added to a candidate child node, the verification process differs from that for candidate child nodes that do not add database hints. This is because these candidate child nodes do not change the existing set of database hints; therefore, there is no need to check for database hint conflicts or changes in the execution plan, as the set of database hints remains unchanged, and so does the execution plan. In other words, candidate child nodes without new database hints are pure reasoning nodes, meaning they only perform reasoning through analysis, problem diagnosis, and strategy planning.

[0147] Therefore, the initial process reward model can be invoked to score the thought process of candidate child nodes that do not have new database prompts, judging their rationality, logic, and guiding value for subsequent optimization. That is, the initial process reward model determines the cumulative exploration value of the candidate child nodes, and then determines whether the cumulative exploration value meets the screening criteria, which can be that the cumulative exploration value is greater than a value threshold. Based on this, if the cumulative exploration value of a candidate child node is less than the value threshold, the candidate child node with a cumulative exploration value less than the value threshold is pruned, and the pruned candidate child node will no longer be considered as an expansion node. Conversely, if the cumulative exploration value of a candidate child node is greater than the value threshold, that is, if the cumulative exploration value meets the screening criteria, this candidate child node is selected as an expansion node.

[0148] In practical applications, the cumulative exploration value can also be used to initialize the extended node, meaning that the extended node has a cumulative exploration value, and the explorable value can be calculated based on the cumulative exploration value of the extended node. To facilitate understanding of how to calculate the explorable value, a calculation example is shown in the following formula (2):

[0149]

[0150]

[0151]

[0152] in, This represents the explorable value of the extended node s. This indicates the balance of exploration weights. This represents the cumulative exploration value of the expanded node s. Explorable value. The explorable value is the expected cumulative reward obtained after performing the corresponding action of the extended node s. It's actually an average value, which is the value that can be explored. This reflects the average reward that can be obtained by continuing the search from the expanded nodes. Cumulative exploration value. The q-value refers to the total reward obtained in all simulations starting from the extended node s. This means that a reward is obtained for each simulation (i.e., a complete deduction from the extended node s to the terminal node), and these rewards are accumulated into the q-value. The accumulated exploration value of each node is updated during the backtracking phase. The balanced exploration weight records the number of times the extended node s is visited. This means that each time, starting from the root node, each node along the tree is visited once, selecting child nodes down to the leaf nodes. The balanced exploration weight of the visited nodes is updated during the simulation and backtracking phases.

[0153] Therefore, in the case of an initial process reward model, the score of the extended node s can be used as the cumulative exploration value of the extended node s based on the initial process reward model. If there is no initial process reward model, the cumulative exploration value of the extended node s is 0, and the explorable value of the extended node s is also initialized to 0.

[0154] The methods described in the foregoing embodiments allow for pruning of some candidate child nodes, with the retained candidate child nodes serving as expansion nodes. The above describes the method for selecting expansion nodes. After obtaining expansion nodes, the server needs to select one from multiple expansion nodes for inference simulation until a termination node is obtained. As shown in the foregoing embodiments, when the initial process reward model is invoked, the obtained expansion nodes can possess both accumulated exploration value and explorable value. However, when the initial process reward model is not invoked, both the accumulated exploration value and explorable value of the obtained expansion nodes are initialized to 0. Therefore, the following will describe how to select expansion nodes for inference simulation:

[0155] In one exemplary embodiment, performing inference simulation for multiple extended nodes until a termination node is obtained includes: when each extended node has no cumulative exploration value, randomly selecting a target extended node from the multiple extended nodes, and performing inference simulation based on the target extended node until a termination node is obtained; when each extended node has cumulative exploration value, selecting a target extended node from the multiple extended nodes according to the cumulative exploration value, and performing inference simulation based on the target extended node until a termination node is obtained.

[0156] Specifically, when each expansion node has no cumulative exploration value (i.e., both its cumulative and explorable values ​​are initialized to 0), the cumulative and explorable values ​​of multiple expansion nodes are essentially the same. Therefore, a target expansion node for inference simulation is randomly selected from among the multiple expansion nodes, and inference simulation is performed based on the target expansion node until the termination node is obtained. Conversely, when each expansion node has a cumulative exploration value (i.e., the expansion nodes are selected by calling the initial process reward model to obtain the cumulative exploration value), a target expansion node is selected from among the multiple expansion nodes according to its cumulative exploration value. That is, the expansion node with the largest cumulative exploration value is selected as the target expansion node, and inference simulation is performed based on the target expansion node until the termination node is obtained.

[0157] For ease of understanding, the examples shown in the foregoing embodiments will be described, such as... Figure 8As shown, node expansion based on leaf node 801 yields candidate child nodes 802, 803, 804, and 805. After verifying candidate child nodes 802 to 805, pruning is performed on candidate child node 805 according to the verification results. The resulting expanded nodes are candidate child nodes 802, 803, and 804. If all three candidate child nodes have accumulated exploration value, the candidate child node with the largest accumulated exploration value is selected as the target expanded node. If candidate child node 803 has the largest accumulated exploration value, it is selected as the target expanded node. Then, reasoning simulation is performed based on the target expanded node (i.e., candidate child node 803) until the termination node 806 is obtained.

[0158] As described above, a termination node can be either a final termination node or a temporary termination node. A final termination node is a reasoning node where the executed reasoning simulation behavior is a preset behavior. In this embodiment, six reasoning actions are given, and the sixth reasoning action is answering a question. Specifically, the sixth reasoning action indicates that there is no further room for optimization, so the thinking stops and an answer is given. In other words, the sixth reasoning action is the preset behavior. A temporary termination node is a reasoning node that generates a new execution plan.

[0159] Therefore, the following will describe how to perform inference simulation based on target expansion nodes until the termination node is reached to complete the inference. First, we will introduce how to determine the final termination node when the termination node is the final termination node. In an exemplary specific embodiment, inference simulation is performed on multiple expansion nodes until the termination node is obtained, including: performing inference simulation on multiple expansion nodes to obtain multiple inference nodes; if the inference simulation behavior performed by the inference node is a preset behavior, the inference simulation is stopped, and the inference node whose inference simulation behavior is the preset behavior among the multiple inference nodes is selected as the termination node.

[0160] Specifically, during query optimization, the process stops when there's no further room for optimization, indicating that no further optimization will be performed. This lack of optimization space typically occurs in the following situations: 1. Performance has reached the target; 2. All possible optimization directions have been tried; 3. Diminishing returns on optimization; 4. An insurmountable bottleneck has been encountered; 5. Resource constraints; 6. Duplicate or conflicting hints exist. When performance has reached the target, it means the query's performance (e.g., execution time) has met the preset goal, and further improvements through adding hints or adjusting the plan are impossible. When all possible optimization directions have been tried, it means all reasonable hint combinations have been tried, including indexes, join methods, and parallelism, and no new hints offer improvement. When diminishing returns on optimization are observed, it means the benefits of further optimization are very small; for example, multiple attempts may not result in significant performance improvements, or the improvement may be less than a certain threshold. When an insurmountable bottleneck is encountered, it means the query has reached the database's theoretically optimal plan, or is limited by data distribution, hardware resources, etc., preventing further optimization. In cases of resource constraints, it indicates that the optimization process must be stopped due to limitations such as time, computing resources, or memory. In cases of duplicate or conflicting hints, it indicates that when attempting to add a new hint, it was found to conflict with an existing hint, or that it had already been tried without success.

[0161] As explained above, the sixth inference action specifically indicates that there is no room for further optimization, so the thought process stops and an answer is given. In other words, after executing the sixth inference action, the node will stop thinking and give an answer. Therefore, after performing inference simulation on the target expansion node to obtain multiple inference nodes, if the inference simulation behavior performed by the inference node is a preset behavior, the inference simulation stops and an answer is given. The inference node whose inference simulation behavior is the preset behavior is then selected as the termination node, specifically the final termination node. For example, if inference simulation on the target expansion node yields inference nodes A1, A2, and A3, and the inference simulation behavior performed by inference node A2 is the preset behavior, meaning that inference node A2 specifically indicates that there is no room for further optimization, then the inference simulation stops and an answer is given, and inference node A2 is determined as the final termination node.

[0162] The following describes how to determine the method for reaching a temporary termination node when the termination node is a temporary termination node. In an exemplary specific embodiment, inference simulation is performed on multiple extended nodes until a termination node is obtained, including: performing inference simulation on multiple extended nodes to obtain an inference node; when the inference node outputs database prompt information, performing an execution plan query based on the database prompt information to obtain the plan to be verified and the plan identifier of the plan to be verified; when the existing plan identifier and the plan identifier cannot be matched, the inference node that outputs database prompt information among the multiple inference nodes is selected as the termination node.

[0163] Specifically, multiple inference nodes are obtained through inference simulation targeting the extended node. At this point, it's necessary to consider whether the inference nodes output database hints. If so, the database connection is invoked to query the execution plan based on the database hints, obtaining the plan to be verified and the plan identifier matching the plan to be verified. It's then necessary to determine if the obtained plan to be verified is a new execution plan. Therefore, the existing plan identifier of the existing execution plan is obtained, and the existing plan identifier is matched with the plan identifier of the plan to be verified. If the identifier matching result indicates a successful match, it means the plan to be verified is not a new execution plan. Since the temporary termination node is the inference node that generates the new execution plan, the successfully matched inference node is not a temporary termination node. Conversely, if the identifier matching result indicates a failed match, meaning the existing plan identifier and the plan identifier cannot match, the inference node that outputs database hints and fails to match will be designated as the termination node, specifically a temporary termination node.

[0164] In practical applications, determining whether to stop optimization to reach a termination node can also be done in the following ways: In one exemplary optional embodiment, a performance threshold is set, and the execution time of the current query is calculated. If the execution time of the current query is lower than the performance threshold, optimization is stopped to reach a termination node. In another exemplary optional embodiment, a maximum number of iterations is set, and the number of iterations for optimizing the query is calculated. If the number of iterations reaches the maximum number of iterations, optimization is stopped to reach a termination node. In another exemplary optional embodiment, the performance of the execution plan obtained from multiple consecutive iterations is evaluated. If the performance does not increase, optimization is stopped to reach a termination node. In yet another exemplary optional embodiment, if the current inference node has no more effective actions, i.e., the current inference node has no new hints to generate, optimization is stopped to reach a termination node.

[0165] And in practical applications, by selecting the target expansion node By conducting reasoning simulations and considering the cumulative exploration value, the explorable value obtained after initializing the target expansion node can be obtained according to the aforementioned formula (2). However, it is indeed possible to explore value. Guiding node selection to find the correct problem-solving chain is crucial. Therefore, to improve the reliability and accuracy of exploration value, this embodiment can continuously update the explorable value of each inference node based on the actual query performance corresponding to the given final optimizer hint set ("Hint Set"). Specifically, based on the current state of the expanded nodes, there are two cases:

[0166] (1) If the inference node already corresponds to a new execution plan (different from the execution plans that have been explored in the entire inference tree), then the execution plan and query are used to obtain a query latency using the database execution engine, and the cumulative exploration value of the inference node is obtained, and the cumulative exploration value of the inference node corresponds to the range of [-1,1].

[0167] (2) If the inference node does not obtain a new execution plan, it will randomly simulate downwards until it encounters the final termination node or reaches a temporary termination node (as described in the previous embodiment). The execution plan and query will be used to obtain a query latency using the database execution engine, or the plan-query latency cache will be checked, and the cumulative exploration value of the termination node will be obtained. The cumulative exploration value of the termination node corresponds to the range [-1, 1]. The calculation of the cumulative exploration value at this time is as follows: (3)

[0168] (3)

[0169] in, Indicates the termination node The cumulative exploration value, This indicates the latency of the default execution plan for slow queries. This represents the execution time of the current inference node. Formula (3) ensures that a symmetrical and smooth cumulative exploration value is achieved, and the reward range of the cumulative exploration value is [-1, 1].

[0170] Step 306: Based on the termination node, backtrack to the root node that matches the query question sample to obtain the reasoning thought chain, and obtain multiple reasoning thought chains by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0171] Specifically, the server backtracks from the termination node to the root node that matches the query question sample to obtain the reasoning thought chain. Multiple reasoning thought chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation, and backtracking. As described above, after determining the termination node, its explorable value, cumulative exploration value, and balanced exploration weight can be determined. Therefore, during the process of backtracking from the termination node to the root node to obtain the reasoning thought chain, the explorable value, cumulative exploration value, and balanced exploration weight of each node during the backtracking process can be updated. Then, based on the updated explorable value, cumulative exploration value, and balanced exploration weight, the node reasoning, node expansion, reasoning simulation, and backtracking processes are repeatedly executed in the manner described in the aforementioned embodiments to obtain multiple reasoning thought chains. Since there may be multiple different termination nodes during the repeated execution of node reasoning, node expansion, reasoning simulation, and backtracking, each backtracking based on a different termination node can update the value and weight, and obtain the reasoning thought chain obtained from the termination node to the root node in each backtracking. Multiple reasoning thought chains can be obtained by repeating the aforementioned steps multiple times. Therefore, it can be seen that the multiple reasoning chains obtained all include the root node that matches the query question sample, as well as the termination node obtained in accordance with the method described in the aforementioned embodiment. The termination nodes of the multiple reasoning chains can be the same or different, which is not limited here.

[0172] For ease of understanding, Figure 8 The example shown will be further explained, such as... Figure 9 As shown, starting from the terminating node 901, we can backtrack upwards sequentially, passing through nodes 902, 903, and 904 until reaching the root node 905, which matches the query sample. The resulting reasoning chain is: root node 905—node 904—node 903—node 902—node 901. Then, we update the explorable value, cumulative exploration value, and balanced exploration weight of root node 905, node 904, node 903, node 902, and node 901 respectively. By repeating node reasoning, node expansion, reasoning simulation, and backtracking from root node 905 in a similar manner as described above, we can obtain multiple reasoning chains.

[0173] To facilitate understanding of how multiple reasoning chains are acquired, based on Figure 9 Further examples are provided, such as Figure 10 As shown, based on Figure 10As shown in Part (A), completing one node reasoning, node expansion, and reasoning simulation from root node 1001 yields terminal node 1002. Backtracking is then performed from terminal node 1002 back to root node 1001 to sequentially update the explorable value, cumulative exploration value, and balanced exploration weight of terminal node 1002, node 1003, node 1004, node 1005, and root node 1001. At this point, a reasoning chain is obtained, connecting root node 1001 to nodes 1003, 1004, 1005, and finally to terminal node 1002. Then, based on... Figure 10 As shown in Part (B), root node 1001, based on the updated explorable value, cumulative exploration value, and balanced exploration weight of each node, completes another node reasoning, node expansion, and reasoning simulation from root node 1001. If a termination node 1006 is obtained at this point, backtracking is performed from termination node 1006 to root node 1001 to sequentially update the explorable value, cumulative exploration value, and balanced exploration weight of termination node 1007, node 1008, and root node 1001. At this point, another reasoning chain is obtained, connecting root node 1001 to nodes 1007, 1008, and then to termination node 1006. Figure 10 It can be seen that each reasoning chain includes a root node that matches the query question sample, and a terminal node obtained by repeatedly executing node reasoning, node expansion, and reasoning simulation starting from the root node.

[0174] Step 308: Construct training samples based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0175] The query plan is a series of operational steps generated by the query optimizer to execute an SQL query. The query plan includes at least the order of operations, which indicates the execution order of each query action (i.e., the inference actions matched by nodes in this embodiment). In other words, the target inference model can derive the corresponding operation order (i.e., the corresponding output inference chain) based on the query question. Furthermore, in practical applications, the query plan may also include, but is not limited to, access methods and join algorithms. Access methods indicate how to access data in a table (e.g., full table scan, index scan), and join algorithms indicate how to connect tables (e.g., nested loop join, hash join, sort-merge join, etc.). In other words, the target inference model can also output access methods and join algorithms based on the query question.

[0176] Specifically, as described above, all paths ending at either a temporary or final termination node can be considered valid reasoning chains. For each reasoning chain, corresponding training samples are constructed using the query question samples. For example, there may be multiple reasoning chains. And query question samples Then the constructed training samples are ( (), ), ..., ( The aforementioned training samples are used to fine-tune the initial inference model to obtain the target inference model. Specifically, based on the training samples, a predicted query plan is obtained from the initial inference model. Then, a first model loss is constructed using the predicted query plan and the training samples. This first model loss is then used to fine-tune the initial inference model to obtain the target inference model. This target inference model is then used to perform question reasoning based on the query question to obtain the corresponding query plan. The implementation method for model training is described in detail in subsequent embodiments.

[0177] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0178] In the training sample construction method of the above model, node reasoning on the query question sample can gradually select leaf nodes that meet the query question requirements starting from the root node, and expand the leaf nodes to ensure the completeness of node exploration. By backtracking from the terminating node to the root node, multiple reasoning chains are obtained as needed. This allows for the gradual focus on more optimized reasoning chains through high-value nodes. By constructing training samples using reasoning chains and query question samples, the initial reasoning model is fine-tuned so that it can learn from the more optimized reasoning chains how to obtain reasoning nodes and reasoning paths more accurately and efficiently. As a result, the query plan output by the target reasoning model obtained through training is more in line with the query question requirements, thereby improving the model's query performance and thus enhancing the accuracy of reasoning queries based on the target reasoning model.

[0179] As described above, all paths ending at temporary or final termination nodes can be considered valid reasoning chains. However, training sample construction needs to consider whether the reasoning chain allows the model to better learn the reasoning path. In other words, to ensure the reliability of model training, high-quality reasoning chains can be selected from valid ones for training sample construction. The following details the methods for selecting reasoning chains. In an exemplary embodiment, such as... Figure 11 As shown, the method for constructing training samples for the model also includes steps 1102 and 1104. Wherein:

[0180] Step 1102: Determine the explorable value, cumulative exploration value, and balanced exploration weight of the termination node. In the process of backtracking from the termination node to the root node that matches the query question sample, update the explorable value of each node in the backtracking process according to the explorable value, cumulative exploration value, and balanced exploration weight of the termination node.

[0181] Specifically, as can be seen from the aforementioned embodiments, after determining the termination node through the aforementioned reasoning simulation process, the execution plan of the termination node can be used for reward calculation, that is, determining the explorable value, cumulative exploration value, and balanced exploration weight of the termination node. In the process of backtracking from the termination node to the root node that matches the query question sample, the explorable value of each node in the backtracking process is updated according to the explorable value, cumulative exploration value, and balanced exploration weight of the termination node. That is, the explorable value, cumulative exploration value, and balanced exploration weight of each node in the reasoning thought chain are updated from bottom to top. In fact, the original cumulative exploration value of any node in the reasoning thought chain is updated according to the cumulative exploration value of the termination node, thereby obtaining the cumulative exploration value of any node in the reasoning thought chain. To facilitate understanding of the aforementioned update method, an example of updating the cumulative exploration value is shown in the following formula (4):

[0182] (4)

[0183] in, Represents a node The updated cumulative exploration value, Indicates the termination node The cumulative exploration value, Represents a node The existing accumulated exploration value.

[0184] Based on this, the updated balance exploration weight is obtained according to the original balance exploration weight of any node in the reasoning chain. To facilitate understanding of the aforementioned update method, an example of the updated balance exploration weight is shown in the following formula (5):

[0185] (5)

[0186] in, Represents a node The updated balance exploration weights, Represents a node The original balance exploration weight.

[0187] Finally, for any node in the reasoning chain, the updated explorable value can be obtained by applying the updated balanced exploration weight and the updated cumulative exploration value. To facilitate understanding of the aforementioned update method, an example of obtaining the updated explorable value is shown in the following formula (6):

[0188] (6)

[0189] in, Represents a node The updated explorable value, Represents a node The updated cumulative exploration value, Represents a node The updated balance exploration weights.

[0190] Step 1104: Select a sample reasoning chain from multiple reasoning chains based on the cumulative exploration value of the terminal node of the reasoning chain and the explorable value of each node in the reasoning chain.

[0191] Specifically, the server filters multiple reasoning thought chains based on the cumulative exploration value of the terminal node and the explorable value of each node within the reasoning thought chain. This process selects high-quality reasoning thought chains from the effective ones for training sample construction. In other words, reasoning thought chains with both high cumulative exploration value and high explorable value of each node are selected as positive sample thought chains. Conversely, reasoning thought chains with low cumulative exploration value and low explorable value of each node are selected as negative sample thought chains. The following section details how the sample thought chains are selected:

[0192] In one exemplary embodiment, the sample thought chain includes a positive sample thought chain and a training sample constructed from the positive sample thought chain and query question samples, which are used to fine-tune the initial inference model to obtain the target inference model.

[0193] Based on this, according to the cumulative exploration value of the terminal node of the reasoning thought chain and the explorable value of each node in the reasoning thought chain, sample thought chains are selected from multiple reasoning thought chains, including: selecting the first candidate thought chain from multiple reasoning thought chains whose cumulative exploration value of the terminal node satisfies the first screening condition; calculating the average explorable value for each node in each first candidate thought chain, and selecting the first candidate thought chain with the largest average explorable value as the positive sample thought chain.

[0194] The first screening criterion is that the cumulative exploration value falls within a first preset percentage range. This means sorting the terminal nodes by their cumulative exploration value from largest to smallest, and then selecting the reasoning chain to which the terminal node belongs based on the total number of terminal nodes and the value obtained from the first preset percentage. For example, with a first preset percentage of 20%, if there are 10 terminal nodes obtained through the aforementioned embodiment, then the first screening criterion is that the cumulative exploration value is among the top 2 (i.e., 20% * 10 = 2). The average explorable value is calculated based on the explorable value of each node and the total number of nodes in the reasoning chain.

[0195] Specifically, based on the cumulative exploration value of the termination nodes in multiple reasoning thought chains, the multiple reasoning thought chains are filtered according to the first screening condition. That is, the reasoning thought chains are sorted from largest to smallest according to the cumulative exploration value of the termination nodes to obtain the sorting result. Then, the number of filtering chains is selected according to the first preset percentage indicated by the first screening condition and the total number of reasoning thought chains. Then, the reasoning thought chains selected from the sorting result in descending order of the number of filtering chains are selected as the first candidate thought chains. For example, taking the above example for further explanation, taking the first preset percentage as 20% as an example, the termination nodes obtained through the above embodiment are 10, that is, 10 reasoning thought chains can be constructed at this time. Then, the first screening condition is that the value of the cumulative exploration value is in the top 2 (i.e., 20% * 10 = 2). If the 10 reasoning thought chains are reasoning thought chain B1, reasoning thought chain B2 to reasoning thought chain B10, and the cumulative exploration value of the termination nodes in reasoning thought chain B1 and the cumulative exploration value of the termination nodes in reasoning thought chain B2 are in the top 2, then reasoning thought chain B1 and reasoning thought chain B2 are selected as the first candidate thought chains.

[0196] Furthermore, for each node in each first candidate thought chain, the average explorable value of the first candidate thought chain is calculated. Then, based on the average explorable value of the first candidate thought chains, a selection is made, that is, the first candidate thought chain with the largest average explorable value is selected as the positive sample thought chain. For example, inference thought chain B1, which consists of inference nodes A1, A2, and A3, if the explorable value of inference node A1 is 0.8, the explorable value of inference node A2 is 0.6, and the explorable value of inference node A3 is 0.4, then the average explorable value of inference thought chain B1 is 0.6. Inference thought chain B2, which consists of inference nodes A1, A4, A5, and A6, if the explorable value of inference node A1 is 0.8, the explorable value of inference node A4 is 0.7, the explorable value of inference node A5 is 0.6, and the explorable value of inference node A6 is 0.5, then the average explorable value of inference thought chain B2 is 0.65. Since the average explorable value of reasoning chain B2 is greater than that of reasoning chain B1 (i.e., 0.65 is greater than 0.6), reasoning chain B2 is then considered a positive sample reasoning chain.

[0197] In an exemplary alternative embodiment, the sample thought chain further includes a negative sample thought chain. Based on this, the training samples can include positive sample thought chains and query question samples, or they can include negative sample thought chains and query question samples. In this case, the training samples are specifically used to fine-tune the initial process reward model to obtain a target process reward model. This allows the obtained target process reward model to be used to assign cumulative exploration value to the expanded nodes during the node expansion process based on the query question.

[0198] Based on this, the training sample construction method of the model also includes: selecting the second candidate thought chain from multiple reasoning thought chains whose cumulative exploration value of the termination node satisfies the second screening condition; calculating the average explorable value for each node in each second candidate thought chain, and selecting the second candidate thought chain with the smallest average explorable value as the negative sample thought chain.

[0199] The second screening criterion is that the cumulative exploration value falls within a second preset percentage range. This means sorting the terminal nodes by their cumulative exploration value from smallest to largest, and then selecting the reasoning chain to which the terminal node belongs, based on the total number of terminal nodes selected from smallest to largest and the value obtained from the second preset percentage. For example, with a second preset percentage of 20%, if there are 10 terminal nodes obtained through the aforementioned embodiment, then the second screening criterion is that the cumulative exploration value falls within the bottom 2 (i.e., 20% * 10 = 2). The average explorable value is calculated based on the explorable value of each node and the total number of nodes in the reasoning chain.

[0200] Specifically, based on the cumulative exploration value of the termination nodes in multiple reasoning thought chains, the multiple reasoning thought chains are filtered according to the second screening condition. That is, the reasoning thought chains are sorted from smallest to largest according to the cumulative exploration value of the termination nodes to obtain the sorting result. Then, the number of filtering chains is selected according to the second preset percentage indicated by the second screening condition and the total number of reasoning thought chains. Then, the reasoning thought chains selected from the sorting result from smallest to largest are selected as the second candidate thought chains. For example, taking the above example for further explanation, taking the second preset percentage as 20% as an example, the termination nodes obtained through the above embodiment are 10, that is, 10 reasoning thought chains can be constructed at this time. Then, the second screening condition is that the value of the cumulative exploration value is in the last 2 (i.e., 20% * 10 = 2). If the 10 reasoning thought chains are reasoning thought chain B1, reasoning thought chain B2 to reasoning thought chain B10, and the cumulative exploration value of the termination node in reasoning thought chain B9 and the cumulative exploration value of the termination node in reasoning thought chain B10 are in the last 2, then reasoning thought chain B9 and reasoning thought chain B10 are selected as the second candidate thought chains.

[0201] Furthermore, for each node in each second candidate thought chain, the average explorable value of the second candidate thought chain is calculated. Then, based on the average explorable value of the second candidate thought chains, a selection is made, that is, the second candidate thought chain with the smallest average explorable value is selected as the negative sample thought chain. For example, reasoning thought chain B9 consists of reasoning nodes A7, A8, and A9. If the explorable value of reasoning node A7 is 0.2, the explorable value of reasoning node A8 is 0.25, and the explorable value of reasoning node A9 is 0.3, then the average explorable value of reasoning thought chain B9 is 0.25. The reasoning chain B10 consists of reasoning nodes A7, A10, A11, and A12. If the explorable value of reasoning node A7 is 0.3, the explorable value of reasoning node A10 is 0.3, the explorable value of reasoning node A11 is 0.4, and the explorable value of reasoning node A12 is 0.4, then the average explorable value of reasoning chain B10 is 0.35. Since the average explorable value of reasoning chain B9 is less than the average explorable value of reasoning chain B10 (i.e., 0.25 is less than 0.35), reasoning chain B9 is considered a negative sample reasoning chain.

[0202] Based on this, step 308 involves constructing training samples using the query question samples and multiple reasoning chains, specifically including:

[0203] Step 1106: Construct training samples based on the query question samples and the sample thought chain.

[0204] Specifically, training samples are constructed based on query question samples and sample thought chains. This means constructing training samples based on positive sample thought chains and query question samples, or based on query question samples and positive sample thought chains, and query question samples and negative sample thought chains. For example, there may be multiple reasoning thought chains. And query question samples And the reasoning chain The positive sample thinking chain, and the reasoning thinking chain. If the thought chain is negative, then the constructed training samples can be ( The constructed training samples may also include ( )as well as( ).

[0205] In one exemplary embodiment, the training samples include: query question samples and positive sample thought chains, and query question samples and negative sample thought chains. That is, as described in the foregoing embodiments, positive and negative sample thought chains are selected from multiple reasoning thought chains.

[0206] Based on this, the training samples are also used to fine-tune the initial process reward model to obtain the target process reward model. The target process reward model is used to assign cumulative exploration value to the expanded nodes during the node expansion process based on the query question. The aforementioned training samples can also be used to fine-tune the initial process reward model to obtain the target process reward model, that is, to obtain the predicted cumulative exploration value of each node during the question reasoning process based on the query question samples through the initial process reward model; a second model loss is constructed based on the predicted cumulative exploration value of each node and the training samples, and the process reward model is fine-tuned using the second model loss to obtain the target process reward model. The implementation method for model training is described in detail in subsequent embodiments.

[0207] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0208] In this embodiment, considering whether the reasoning thought chain can enable the model to better learn reasoning paths, the performance quality of the reasoning thought chain is evaluated by accumulating exploration value and explorable value. High-quality reasoning thought chains are selected from the effective ones for constructing training samples, ensuring the reliability of the constructed training samples and thus improving the reliability of the model obtained from training based on these samples. Furthermore, for different model training needs, positive and negative sample thought chains are selected to improve the data completeness and flexibility of the constructed training samples, further meeting the data requirements of model training and enhancing the reliability of model training.

[0209] The above embodiments describe a method for constructing training samples. As can be seen from the foregoing embodiments, the constructed training samples are used to fine-tune the inference model and the process reward model to obtain the target inference model and the target process reward model. Therefore, the following specific embodiments illustrate how to obtain the target inference model and the target process reward model: In an exemplary embodiment, such as... Figure 12 As shown, a training method for an inference model is provided, which can be applied to... Figure 1 Taking server 104 as an example, it can be understood that this method can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. In this embodiment, the method includes steps 1202 to 1206. Wherein:

[0210] Step 1202: Obtain the training samples constructed using the model's training sample construction method.

[0211] Specifically, the training samples required for model training can be obtained through the training sample construction method described in the foregoing embodiments. The training samples include at least query question samples and positive sample thought chains, and when optimization of the query process based on a process reward model is required, the training samples also include query question samples and negative sample thought chains. For example, as illustrated in the foregoing examples, there may be multiple reasoning thought chains. And query question samples And the reasoning chain The positive sample thinking chain, and the reasoning thinking chain. If the thought chain is negative, then the constructed training samples can be ( The constructed training samples may also include ( )as well as( ).

[0212] Step 1204: Obtain the predicted query plan based on the training samples using the initial inference model.

[0213] Specifically, the server obtains a predicted query plan based on the training samples using an initial inference model. That is, the server uses the initial inference model to perform query inference on the query question samples in the training samples, thereby obtaining a predicted query plan for the query question samples. In a manner similar to the aforementioned embodiments, the initial inference model performs node inference on the query question samples in the training samples to obtain leaf nodes, then expands the leaf nodes to obtain extended nodes, and performs inference simulation on the extended nodes until a terminating node is obtained. From the terminating node, it sequentially backtracks to the root node matching the query question sample to obtain the predicted thought chain, and the predicted query plan matching the predicted thought chain.

[0214] In practical applications, training samples, including query question samples and positive sample thought chains, are split into multiple training sub-samples. This means they are split according to the nodes in the positive sample thought chains. The input to the initial inference model for each training sub-sample is the current state, which specifically includes the query question sample corresponding to a node in the positive sample thought chain and the inference text of that node. The output of the initial inference model is the predicted inference text for the next state. This process continues until the predicted inference text for the terminal node is obtained. The predicted query plan for the query question sample can then be the predicted inference text for the terminal node, or it can specifically include the predicted inference text output by the initial inference model for each training sub-sample.

[0215] Step 1206: Construct the first model loss according to the predicted query plan and training samples, and fine-tune the initial inference model using the first model loss to obtain the target inference model; the target inference model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0216] Specifically, the server constructs the first model loss based on the predicted query plan and the training samples. That is, the server constructs the first model loss based on the query plan that matches the positive sample thought chains in the training samples. As mentioned earlier, the predicted query plan for a query question sample can also include the predicted inference text output by the initial inference model for each training subsample. Since the positive sample thought chains in the training samples include inference text corresponding to multiple nodes, the first model loss can also be constructed based on the predicted inference text output by each training subsample and the inference text corresponding to each node in the positive sample thought chains. The method of constructing the first model loss is not limited here.

[0217] Based on this, the initial inference model is fine-tuned using the first model loss to obtain the target inference model. Specifically, the model parameters of the initial inference model are adjusted based on the first model loss. First, the convergence condition of the initial inference model's loss function is determined using the first model loss. If the convergence condition is not met, the model parameters of the initial inference model are adjusted using the first model loss. This process continues until the initial inference model's loss function reaches convergence. Then, the target inference model is obtained based on the model parameters obtained after the last adjustment. In practical applications, the trained target inference model is used to derive the corresponding query plan based on the query question.

[0218] When it is necessary to optimize the query process based on a process reward model, that is, to train the required process reward model based on the constructed training samples, the following describes this process: In an exemplary optional embodiment, the training samples include query question samples. Specifically, the training samples include: query question samples and positive sample thought chains, and query question samples and negative sample thought chains.

[0219] Based on this, the training method for the inference model also includes: obtaining the cumulative exploration value of each node during the problem reasoning process based on the query question sample through the initial process reward model; constructing a second model loss according to the cumulative exploration value of each node and the training sample, and fine-tuning the process reward model through the second model loss to obtain the target process reward model; the target process reward model is used to assign the cumulative exploration value of the obtained extended nodes during the node expansion process of problem reasoning based on the query question.

[0220] Specifically, the server obtains the cumulative exploration value of predictions for each node during the problem reasoning process based on the query problem sample through the initial process reward model. Furthermore, the prediction header of the next token in the initial process reward model is replaced with a scalar value header (containing a linear layer and a tanh function) to limit the output range of the cumulative exploration value of predictions for each node of the initial process reward model to [-1, 1].

[0221] Then, the second model loss is constructed based on the cumulative exploration value predicted by each node and the training samples. That is, the required second model loss is constructed based on the cumulative exploration value predicted by each node, the cumulative exploration value of each node in the positive sample thought chain in the training samples, and the cumulative exploration value of each node in the negative sample thought chain in the training samples. The specific loss function refers to the standard Bradley-Terry model. For ease of understanding how to calculate the second model loss, the calculation example is shown in the following formula (7):

[0222] (7)

[0223] in, This represents the loss of the second model. Indicates an intermediate state. This indicates a query for a sample question. This represents the thought process from step 1 to step i. This represents the positive sample thought chain from step 1 to step i. This represents the negative sample thought chain from step 1 to step i. This represents the cumulative exploration value (i.e., score) of the prediction for the i-th step of the positive sample thinking chain. This represents the cumulative exploration value (i.e., score) of the prediction at the i-th step of the negative sample thought chain.

[0224] Based on this, the initial process reward model is fine-tuned using the second model loss to obtain the target process reward model. Specifically, the model parameters of the initial process reward model are adjusted based on the second model loss. First, the convergence condition of the loss function of the initial process reward model is determined using the second model loss. If the convergence condition is not met, the model parameters of the initial process reward model are adjusted using the second model loss. This process continues until the loss function of the initial process reward model reaches convergence. Then, the target process reward model is obtained based on the model parameters obtained after the last adjustment. In practical applications, this trained target process reward model is used to assign cumulative exploration value to the expanded nodes during the node expansion process based on the query question.

[0225] In practical applications, to better construct training samples and train a better general query optimizer inference model, a multi-stage training method can be used for model training. First, the inference model is initialized, which is to generate high-quality inference thought chains when there is no process reward model. Since the process reward model only uses the terminal node query execution verification to obtain the explorable value (i.e., Q-value) of each node, following a similar approach described in the previous embodiments, multiple inference thought chains are obtained for query question samples. Then, positive sample thought chains are selected, and the initial inference model is fine-tuned to the first inference model LLM-r1 using query question samples and positive sample thought chains. Finally, the initial process reward model PRM-r1 is fine-tuned by selecting negative sample thought chains.

[0226] Since the first-process reward model PRM-r1 can assign cumulative exploration value to the expanded nodes during the node expansion process based on the query question, that is, by using the training sample construction method of the first inference model LLM-r1 and the first-process reward model PRM-r1 to construct higher-quality training samples, the first-process reward model PRM-r1 significantly improves the quality of the training samples and covers more query patterns. The resulting training samples can then be used to fine-tune the first inference model LLM-r1 and the first-process reward model PRM-r1 again, resulting in the second inference model LLM-r2 and the second-process reward model PRM-r2. Considering the possibility of unresolved issues, namely the lack of reduction in query latency during query optimization, a random seed is used to add expanded nodes. The resulting training samples can then be used to fine-tune the second inference model LLM-r2 and the second-process reward model PRM-r2 again, resulting in the third inference model LLM-r3 and the third-process reward model PRM-r3, which can handle more complex query patterns. Then, the third reasoning model LLM-r3 and the third process reward model PRM-r3 are used as the target reasoning model and the target process reward model in the process of reasoning for practical problems.

[0227] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0228] In the training method of the above reasoning model, the initial reasoning model is fine-tuned by constructing training samples through reasoning thought chains and query question samples. This allows the initial reasoning model to learn how to obtain reasoning nodes and reasoning paths more accurately and efficiently from a more optimized reasoning thought chain. As a result, the query plan output by the target reasoning model obtained through training is more in line with the requirements of the query question. This improves the query performance of the model and thus enhances the accuracy of reasoning queries based on the target reasoning model.

[0229] The above embodiments describe the methods for constructing training samples and training models. As can be seen from the foregoing embodiments, the obtained target reasoning model and target process reward model are used for problem reasoning. Therefore, the following specific embodiments illustrate how to perform problem reasoning: In an exemplary embodiment, such as... Figure 13 As shown, a problem reasoning method is provided, which can be applied to... Figure 1 Taking server 104 as an example, it can be understood that this method can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. In this embodiment, the method includes steps 1302 to 1306. Wherein:

[0230] Step 1302: Obtain the query question and perform node reasoning on the query question based on the target reasoning model obtained through the training method of the reasoning model to obtain the leaf node.

[0231] Specifically, the server first obtains the query question; that is, the server can obtain the query question in real time, or it can obtain the query question from the data storage system, which will not be specified here. Based on this, since the target inference model can be obtained through the aforementioned embodiments, the leaf nodes are obtained by performing node inference on the query question through the target inference model. The method of obtaining the leaf nodes by node inference is similar to that in the aforementioned embodiments and will not be repeated here.

[0232] Step 1304: Expand the leaf nodes to obtain extended nodes, and perform reasoning simulation on the extended nodes until the termination node is obtained.

[0233] Specifically, the server continues to expand the leaf nodes using the target inference model to obtain extended nodes, and then performs inference simulation on the extended nodes until the terminal node is obtained. If the process reward model is not considered at this point, i.e., prompt words are constructed based on the node state matched by the leaf node and the action guidance information matched by different inference actions, the target inference model generates multiple candidate extended nodes based on the prompt words, and then randomly selects one of the multiple candidate extended nodes as the extended node.

[0234] In one exemplary embodiment, expanding a leaf node to obtain an expanded node includes: expanding a leaf node to obtain multiple candidate expanded nodes, determining the cumulative exploration value of each candidate expanded node through a target process reward model, and selecting the candidate expanded node with the largest cumulative exploration value as the expanded node.

[0235] Specifically, considering the process reward model, prompt words are constructed based on the node states matched by leaf nodes and the action guidance information matched by different inference actions. The target inference model generates multiple candidate expansion nodes based on the prompt words. Then, the cumulative exploration value of each candidate expansion node is determined through the target process reward model, and the candidate expansion node with the largest cumulative exploration value is selected as the expansion node. For ease of understanding, the leaf nodes obtained in the aforementioned embodiment are further described, such as... Figure 14 As shown, by expanding the node according to the node state matched by leaf node 1401 and various action guidance information, candidate child nodes 1402, 1403, 1404 and 1405 can be obtained. Figure 14 Part (A) shows the method of randomly selecting the expansion node, that is, randomly selecting candidate child node 1402 as the expansion node from candidate child nodes 1402, 1403, 1404, and 1405. Figure 14 Part (B) shows the method of selecting expansion nodes based on cumulative exploration value. That is, the cumulative exploration value of candidate child node 1402, candidate child node 1403, candidate child node 1404 and candidate child node 1405 are first determined by the target process reward model. If the cumulative exploration value of candidate child node 1403 is the largest, then candidate child node 1403 is selected as the expansion node.

[0236] Furthermore, the server performs reasoning simulation on the extended nodes using the target reasoning model until the termination node is obtained. That is, the query optimization stops when the final termination node or a temporary termination node is obtained. The method for determining whether the termination node has been reached is similar to that described in the previous embodiments, and will not be repeated here.

[0237] Step 1306: Based on the termination node, backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

[0238] Specifically, by backtracking sequentially from the terminating node to the root node that matches the query question, a query plan corresponding to the query question is obtained. In other words, by backtracking sequentially upwards along the reasoning tree from the terminating node to the root node that matches the query question, the resulting reasoning chain matching query plan is the query plan corresponding to the query question. For ease of understanding, as follows... Figure 15 As shown, node reasoning and node selection using the query question as the root node yield leaf nodes. Expanding the leaf nodes generates multiple candidate child nodes. After verifying these candidate child nodes, pruning is performed. The cumulative exploration value of each candidate expansion node is determined using the objective process reward model. The candidate expansion node with the highest cumulative exploration value is selected as the expansion node. Reasoning simulation is then performed based on the expansion node until a termination node is obtained. Finally, the root node matching the query question is backtracked through the termination node to obtain the query plan corresponding to the query question.

[0239] To further understand how a query plan is obtained in practical applications, the following example of the input query question will be used for a detailed explanation:

[0240] select o_year,

[0241] sum(case when nation = 'SAUDI ARABIA' then volume else 0 end) / sum(volume) as mkt_share

[0242] from (

[0243] select extract(year from o_orderdate) as o_year,

[0244] l_extendedprice * (1 - l_discount) as volume,

[0245] n2.n_name as nation

[0246] from part, supplier, lineitem, orders, customer, nation n1, nationn2, region

[0247] where p_partkey = l_partkey

[0248] and s_suppkey = l_suppkey

[0249] and l_orderkey = o_orderkey

[0250] and o_custkey = c_custkey

[0251] and c_nationkey = n1.n_nationkey

[0252] and n1.n_regionkey = r_regionkey

[0253] and r_name = 'AFRICA'

[0254] and s_nationkey = n2.n_nationkey

[0255] and o_orderdate between date '1995-01-01' and date '1996-12-31'

[0256] and p_type = 'ECONOMY POLISHED NICKEL' ) as all_nations

[0257] group by o_year

[0258] order by o_year;

[0259] The aforementioned query corresponds to root node S1. Based on the execution information of root node S1, it can be determined that filtering p_type='ECONOMY POLISHED NICKEL' on the part table resulted in a very small number of filtered rows (0.0056%), but the loop iterated 600,000 times. This indicates that in the join order, part might be placed later, causing a large number of unnecessary rows to be processed before filtering. In other words, there is a problem with the current join order: the current execution plan places the part table later in the join order, resulting in a large number of rows being processed before the filtering conditions are applied. This may be due to the optimizer misestimating the data distribution of each table, leading to a suboptimal join order.

[0260] Therefore, the "raise sub-problem" reasoning action, specifically "raise sub-problem: The current join order is poor, placing the part table at the end; how to correct it?", matches the leaf node S2. Based on this, to address the poor join order, the part table should be placed before the join. JOIN_PREFIX(part) can be used to force the optimizer to process the part table first, and since this hint exists in the candidate hints, the "single-step thinking - correct order" reasoning action should be executed. Therefore, the node matching the "single-step thinking - correct order" reasoning action is the extended node S3.

[0261] Similarly, the filtering in the `region` table (r_name='AFRICA') has only one row, but the join between `region` and `n1` in the execution plan generates five rows (because each region has five nations). Then, a hash join is performed with `orders`, and the estimated number of rows differs significantly from the actual number (estimated 90,000, actual 3.16 million). This may be because the optimizer underestimated the number of rows in the `orders` table that meet the time condition, leading to an inappropriate join order. If `region` and `n1` could be processed first, and then the AFRICA client could be quickly filtered, the amount of data in subsequent joins might be reduced. At this point, the "raise sub-problem" inference action is executed again, specifically "raise sub-problem: The current join order is poor, underestimating the number of rows in the `orders` table that meet the time condition; how to correct it?". The node matched by this second "raise sub-problem" inference action could be node S4. Then, the "single-step thinking - correct order" inference action is executed again, and the node matched by this second "single-step thinking - correct order" inference action could be node S5.

[0262] Based on this, the table scan on orders actually scanned 1.5 million rows, but after filtering, approximately 630,000 rows remained. A full table scan was used here, which may be inefficient. The "raise sub-problem" reasoning action is executed again, specifically "raise sub-problem: The current scan efficiency on the orders table is low, the index settings are improper, how to correct it?". At this point, the node matching the "raise sub-problem" reasoning action is node S6. Then, the "single-step thinking - correct index" reasoning action is executed again, and the node matching the "single-step thinking - correct index" reasoning action is node S7. If the final Hints output is as follows:

[0263] / *+ JOIN_PREFIX(part) INDEX(orders o_orderdate)* /

[0264] <end_of_think>

[0265] At this point, the final termination node S8 is reached, and the sequence is: Root node S1 — Leaf node S2 — Extended node S3 — Node S4 — Node S5 — Node S6 — Node S7 — Termination node S8. The resulting query plan is:

[0266] <answer>

[0267] / *+ JOIN_PREFIX(part) INDEX(orders o_orderdate)* /

[0268] <end_of_answer>

[0269] <Final rewritten query (corresponding to an execution plan)>

[0270] select / *+ JOIN_PREFIX(part) INDEX(orders o_orderdate)* / o_year,

[0271] sum(case when nation = 'SAUDI ARABIA' then volume else 0 end) / sum(volume) as mkt_share

[0272] from (

[0273] select extract(year from o_orderdate) as o_year,

[0274] l_extendedprice * (1 - l_discount) as volume,

[0275] n2.n_name as nation

[0276] from part, supplier, lineitem, orders, customer, nation n1, nation n2, region

[0277] where p_partkey = l_partkey

[0278] and s_suppkey = l_suppkey

[0279] and l_orderkey = o_orderkey

[0280] and o_custkey = c_custkey

[0281] and c_nationkey = n1.n_nationkey

[0282] and n1.n_regionkey = r_regionkey

[0283] and r_name = 'AFRICA'

[0284] and s_nationkey = n2.n_nationkey

[0285] and o_orderdate between date '1995-01-01' and date '1996-12-31'

[0286] and p_type = 'ECONOMY POLISHED NICKEL' ) as all_nations

[0287] group by o_year

[0288] order by o_year;

[0289] It is understood that the foregoing examples are for the purpose of understanding this solution and should not be construed as specific limitations on this solution.

[0290] In the above-mentioned problem reasoning method, the query plan output by the target reasoning model obtained through training is more in line with the problem requirements of the query, thereby improving the accuracy of reasoning queries based on the target reasoning model.

[0291] Based on the detailed description of the foregoing embodiments, the complete process of the training sample construction method, the inference model training method, and the problem reasoning method in the embodiments of this application will be described below. In an exemplary embodiment, such as Figure 16 As shown, a method for constructing training samples for a model, a method for training a reasoning model, and a method for problem reasoning are provided, which are then applied to... Figure 1 Taking server 104 as an example, it can be understood that this method can also be applied to a system including terminal 102 and server 104, and implemented through the interaction between terminal 102 and server 104. In this embodiment, the method includes steps 1601 to 1612. Wherein:

[0292] Step 1601: Using the initial inference model obtained from pre-training, the query question sample is used as the root node to search downwards for multiple candidate child nodes; according to the upper confidence bound of each candidate child node, the optimal child node is selected from the multiple candidate child nodes; the leaf node is obtained by continuing to search downwards along the tree structure through the optimal child node.

[0293] Step 1602: Expand the node based on the node state of the leaf node that meets the node expansion condition to obtain multiple candidate child nodes; select the expanded node from the multiple candidate child nodes.

[0294] Step 1603: If each extended node has no cumulative exploration value, randomly select a target extended node from multiple extended nodes and perform reasoning simulation based on the target extended node until the termination node is obtained; if each extended node has cumulative exploration value, select a target extended node from multiple extended nodes according to the cumulative exploration value and perform reasoning simulation based on the target extended node until the termination node is obtained.

[0295] Step 1604: Determine the explorable value, cumulative exploration value, and balanced exploration weight of the termination node.

[0296] Step 1605: Based on the termination node, backtrack sequentially to the root node that matches the query question sample to obtain multiple reasoning chains; in the process of backtracking sequentially to the root node that matches the query question sample based on the termination node, update the explorable value of each node in the backtracking process according to the explorable value of the termination node, the cumulative explorable value, and the balanced exploratory weight.

[0297] Step 1606: From multiple reasoning thought chains, select the first candidate thought chain whose cumulative exploration value of the termination node meets the first screening condition; calculate the average explorable value for each node in each first candidate thought chain, and select the first candidate thought chain with the largest average explorable value as the positive sample thought chain.

[0298] Step 1607: From multiple reasoning thought chains, select the second candidate thought chain whose cumulative exploration value of the termination node satisfies the second screening condition; calculate the average explorable value for each node in each second candidate thought chain, and select the second candidate thought chain with the smallest average explorable value as the negative sample thought chain.

[0299] Step 1608: Based on the query question sample, obtain the predicted query plan through the initial inference model; construct the first model loss according to the predicted query plan and the positive sample thought chain, and fine-tune the initial inference model through the first model loss to obtain the target inference model.

[0300] Step 1609: Obtain the cumulative exploration value of each node in the process of reasoning based on the query question sample through the initial process reward model; construct the second model loss according to the cumulative exploration value of each node, the cumulative exploration value of the corresponding node in the positive sample thinking chain, and the cumulative exploration value of the corresponding node in the negative sample thinking chain; and fine-tune the process reward model through the second model loss to obtain the target process reward model.

[0301] Step 1610: Obtain the query question; perform node reasoning on the query question based on the target reasoning model to obtain the leaf nodes.

[0302] Step 1611: Expand the leaf nodes to obtain multiple candidate expansion nodes, and determine the cumulative exploration value of each candidate expansion node through the target process reward model; select the candidate expansion node with the largest cumulative exploration value as the expansion node.

[0303] Step 1612: Perform reasoning simulation on the extended nodes until the termination node is obtained; based on the termination node, backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

[0304] It should be understood that the specific implementation methods of steps 1601 to 1612 are similar to those of the aforementioned embodiments, and will not be repeated here.

[0305] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0306] Based on the same inventive concept, this application also provides a model training sample construction apparatus for implementing the above-described model training sample construction method. The solution provided by this apparatus is similar to the implementation described in the above-described method. Therefore, the specific limitations in the embodiments of the one or more model training sample construction apparatus provided below can be found in the limitations of the model training sample construction method described above, and will not be repeated here.

[0307] In one exemplary embodiment, such as Figure 17 As shown, a training sample construction device for a model is provided, including: a node inference module 1702, a node expansion and simulation module 1704, a node backtracking module 1706, and a training sample construction module 1708, wherein:

[0308] The node reasoning module 1702 is used to perform node reasoning on the query question sample to obtain multiple leaf nodes using the initial reasoning model obtained through pre-training.

[0309] The node expansion and simulation module 1704 is used to expand at least one leaf node to obtain multiple expanded nodes, and to perform inference simulation on the multiple expanded nodes until the terminal node is obtained.

[0310] The node backtracking module 1706 is used to backtrack from the termination node to the root node that matches the query question sample to obtain the reasoning thought chain, and to obtain multiple reasoning thought chains by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking.

[0311] The training sample construction module 1708 is used to construct training samples based on query question samples and multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

[0312] In an exemplary embodiment, the node reasoning module is specifically used to use the query question sample as the root node to search downwards for multiple candidate child nodes; select the optimal child node from the multiple candidate child nodes according to the upper confidence bound of each candidate child node; and continue searching downwards along the tree structure through the optimal child node to obtain the leaf node.

[0313] In an exemplary embodiment, the node expansion and simulation module is specifically used to expand a node based on the node state of a leaf node that satisfies the node expansion condition, thereby obtaining multiple candidate child nodes; and to select an expanded node from the multiple candidate child nodes.

[0314] In an exemplary embodiment, the node expansion condition characterizes a leaf node that has not been fully explored; wherein, a leaf node that has not been fully explored includes: there are still logical actions that have not generated child nodes of the leaf node; or, the candidate child nodes of the leaf node have not been visited.

[0315] In an exemplary embodiment, the node expansion and simulation module is specifically used to: when a candidate child node has a new database prompt, perform information matching between the new database prompt and the existing database prompt to obtain an information matching result; when the information matching result indicates an information conflict, prune the candidate child node; wherein, an information conflict means that there is an information conflict between the new database prompt and the existing database prompt; select the candidate child node whose information matching result indicates no information conflict as the expansion node; wherein, no information conflict means that there is no information conflict between the new database prompt and the existing database prompt.

[0316] In an exemplary embodiment, the node expansion and simulation module is specifically used to determine the expansion execution plan of the candidate child node and the current execution plan of the leaf node that meets the node expansion conditions when the candidate child node has a new database prompt message; if the expansion execution plan is consistent with the current execution plan, the candidate child node is pruned; and the candidate child node whose expansion execution plan is inconsistent with the current execution plan is selected as the expansion node.

[0317] In an exemplary embodiment, the node expansion and simulation module is specifically used to determine the cumulative exploration value of a candidate child node when there is no new database prompt information for the candidate child node; and to select the candidate child node whose cumulative exploration value meets the screening criteria from multiple candidate child nodes as the expansion node.

[0318] In an exemplary embodiment, the node expansion and simulation module is specifically used to randomly select a target expansion node from multiple expansion nodes when each expansion node has no cumulative exploration value, and to perform reasoning simulation based on the target expansion node until a termination node is obtained.

[0319] In an exemplary embodiment, the node expansion and simulation module is specifically used to select a target expansion node from multiple expansion nodes according to the cumulative exploration value when each expansion node has a cumulative exploration value, and to perform reasoning simulation based on the target expansion node until a termination node is obtained.

[0320] In an exemplary embodiment, the node expansion and simulation module is specifically used to perform inference simulation on multiple expansion nodes to obtain multiple inference nodes; when the inference simulation behavior performed by the inference node is a preset behavior, the inference simulation is stopped, and the inference node whose inference simulation behavior is the preset behavior among the multiple inference nodes is selected as the termination node.

[0321] In an exemplary embodiment, the node expansion and simulation module is specifically used to perform inference simulation on multiple expansion nodes to obtain inference nodes; when the inference node outputs database prompt information, the execution plan is queried based on the database prompt information to obtain the plan to be verified and the plan identifier of the plan to be verified; when the existing plan identifier and the plan identifier cannot be matched, the inference node that outputs database prompt information among the multiple inference nodes is selected as the termination node.

[0322] In an exemplary embodiment, the node backtracking module is further configured to determine the explorable value, cumulative exploration value, and balanced exploration weight of the termination node, and update the explorable value of each node in the backtracking process according to the explorable value, cumulative exploration value, and balanced exploration weight of the termination node in the process of sequentially backtracking to the root node that matches the query question sample; and select a sample thinking chain from multiple reasoning thinking chains according to the cumulative exploration value of the termination node of the reasoning thinking chain and the explorable value of each node in the reasoning thinking chain.

[0323] The training sample construction module is specifically used to construct training samples based on query question samples and sample thought chains.

[0324] In one exemplary embodiment, the sample thought chain includes a positive sample thought chain;

[0325] The node backtracking module is specifically used to select the first candidate thought chain from multiple reasoning thought chains whose cumulative exploration value of the termination node meets the first screening condition; calculate the average explorable value for each node in each first candidate thought chain, and select the first candidate thought chain with the largest average explorable value as the positive sample thought chain.

[0326] In one exemplary embodiment, the sample thought chain also includes a negative sample thought chain;

[0327] The node backtracking module is also used to select the second candidate thought chain from multiple reasoning thought chains whose cumulative exploration value of the termination node meets the second screening condition; calculate the average explorable value for each node in each second candidate thought chain, and select the second candidate thought chain with the smallest average explorable value as the negative sample thought chain.

[0328] In an exemplary embodiment, the training samples include: query question samples and positive sample thought chains, and query question samples and negative sample thought chains; the training samples are also used to fine-tune the initial process reward model to obtain a target process reward model; the target process reward model is used to assign cumulative exploration value to the obtained extended nodes during the node expansion process of problem reasoning based on the query question.

[0329] In one exemplary embodiment, such as Figure 18 As shown, a training device for an inference model is provided, comprising: a training sample acquisition module 1802, an inference prediction module 1804, and a model training module 1806, wherein:

[0330] The training sample acquisition module 1802 is used to acquire training samples constructed by any one of claims 1 to 13;

[0331] The inference prediction module 1804 is used to obtain a predicted query plan based on the training samples and an initial inference model.

[0332] The model training module 1806 is used to construct the first model loss according to the predicted query plan and training samples, and to fine-tune the initial inference model through the first model loss to obtain the target inference model; the target inference model is used to perform question inference based on the query question to obtain the corresponding query plan.

[0333] In one exemplary embodiment, the training samples include query question samples;

[0334] The model training module is also used to obtain the cumulative exploration value of each node in the process of problem reasoning based on query question samples through the initial process reward model; to construct a second model loss according to the cumulative exploration value of each node and the training samples, and to fine-tune the process reward model through the second model loss to obtain the target process reward model; the target process reward model is used to assign the cumulative exploration value to the obtained extended nodes in the process of node expansion based on query question reasoning.

[0335] In one exemplary embodiment, such as Figure 19 As shown, a problem reasoning device is provided, including: a node reasoning module 1902, a node expansion module 1904, and a problem reasoning module 1906, wherein:

[0336] The node reasoning module 1902 is used to obtain the query question and perform node reasoning on the query question based on the target reasoning model obtained by any one of claims 14 or 15 to obtain leaf nodes;

[0337] The node expansion module 1904 is used to expand leaf nodes to obtain expanded nodes, and to perform inference simulation on the expanded nodes until the terminal node is obtained.

[0338] The problem reasoning module 1906 is used to backtrack sequentially to the root node that matches the query problem based on the termination node, so as to obtain the query plan corresponding to the query problem.

[0339] In an exemplary embodiment, the node expansion module is specifically used to expand the leaf nodes to obtain multiple candidate expansion nodes, and to determine the cumulative exploration value of each candidate expansion node through the target process reward model; and to select the candidate expansion node with the largest cumulative exploration value as the expansion node.

[0340] In one exemplary embodiment, a computer device is provided, which can be a server or a terminal. This embodiment uses a server as an example for description, and its internal structure diagram can be as follows: Figure 20 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The database stores training samples, inference models, and process reward models, among other data related to the embodiments of this application. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for constructing training samples for a model, a method for training an inference model, and a problem reasoning method.

[0341] Those skilled in the art will understand that Figure 20 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0342] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0343] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0344] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0345] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0346] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, etc., and are not limited to these.

[0347] The technical features in the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0348] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.< / answer>

Claims

1. A method for constructing training samples for a model, characterized in that, The method includes: Using the initial inference model obtained through pre-training, multiple leaf nodes are obtained by performing node inference on the query question sample; For at least one of the leaf nodes, node expansion is performed to obtain multiple extended nodes, and inference simulation is performed on the multiple extended nodes until a termination node is obtained; Based on the termination node, backtracking to the root node that matches the query question sample yields a reasoning thought chain, and multiple reasoning thought chains are obtained by repeatedly executing node reasoning, node expansion, reasoning simulation, and backtracking. Training samples are constructed based on the query question samples and the multiple reasoning chains. The training samples are used to fine-tune the initial reasoning model to obtain the target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

2. The method according to claim 1, characterized in that, The node reasoning process on the query question sample yields multiple leaf nodes, including: Using the query sample as the root node, search downwards for multiple candidate child nodes; Select the optimal child node from the plurality of candidate child nodes according to the upper confidence bound of each candidate child node; The leaf node is obtained by continuing the search downwards along the tree structure using the optimal child node.

3. The method according to claim 1, characterized in that, The process of expanding at least one leaf node to obtain multiple expanded nodes includes: Based on the node states of leaf nodes that meet the node expansion conditions, node expansion is performed to obtain multiple candidate child nodes. Select an expansion node from the plurality of candidate child nodes.

4. The method according to claim 3, characterized in that, The node expansion condition indicates that the leaf node has not been fully explored; The leaf node not being fully explored includes: there are still logical actions but no child nodes of the leaf node have been generated; or, the candidate child nodes of the leaf node have not been visited.

5. The method according to claim 3, characterized in that, The step of selecting an expansion node from the plurality of candidate child nodes includes: If the candidate child node has a new database prompt, the new database prompt is matched with the existing database prompt to obtain the information matching result; If the information matching result indicates that there is an information conflict, the candidate child node is pruned; wherein, the information conflict is that there is an information conflict between the new database prompt information and the existing database prompt information; The candidate child node indicating no information conflict is selected as the extended node; wherein, no information conflict means that there is no information conflict between the new database prompt information and the existing database prompt information.

6. The method according to claim 3, characterized in that, The step of selecting an expansion node from the plurality of candidate child nodes includes: If the candidate child node has a new database prompt, determine the extended execution plan of the candidate child node and compare it with the current execution plan of the leaf node that meets the node extension conditions; If the extended execution plan is consistent with the current execution plan, the candidate child nodes are pruned. Select the extended execution plan and the candidate child nodes that are inconsistent with the current execution plan as extended nodes.

7. The method according to claim 3, characterized in that, The step of selecting an expansion node from the plurality of candidate child nodes includes: If no new database prompt information is found for the candidate child node, the cumulative exploration value of the candidate child node is determined. From the multiple candidate child nodes, select the candidate child node whose cumulative exploration value meets the screening criteria as the expansion node.

8. The method according to claim 1, characterized in that, Inference simulations are performed on the multiple extended nodes until a termination node is obtained, including: If each of the aforementioned extended nodes has no cumulative exploration value, a target extended node is randomly selected from the plurality of extended nodes, and reasoning simulation is performed based on the target extended node until a termination node is obtained; If each of the aforementioned extended nodes has accumulated exploration value, a target extended node is selected from the plurality of extended nodes according to the accumulated exploration value, and reasoning simulation is performed based on the target extended node until a termination node is obtained.

9. The method according to claim 1, characterized in that, The process of performing inference simulations on the multiple extended nodes until a termination node is obtained includes: Inference nodes are obtained by performing inference simulations on the multiple extended nodes; If the inference node outputs database prompt information, an execution plan query is performed based on the database prompt information to obtain the plan to be verified and the plan identifier of the plan to be verified. If the existing plan identifier cannot be matched with the plan identifier, the inference node that outputs database prompt information among the multiple inference nodes will be selected as the termination node.

10. The method according to claim 1, characterized in that, The method further includes: The explorable value, cumulative exploration value, and balanced exploration weight of the termination node are determined. During the process of backtracking sequentially to the root node that matches the query question sample based on the termination node, the explorable value of each node in the backtracking process is updated according to the explorable value, cumulative exploration value, and balanced exploration weight of the termination node. Based on the cumulative exploration value of the terminal node of the reasoning chain and the explorable value of each node in the reasoning chain, a sample reasoning chain is selected from the multiple reasoning chains. The step of constructing training samples based on the query question samples and the multiple reasoning chains includes: Training samples are constructed based on the query question samples and the sample thought processes.

11. The method according to claim 10, characterized in that, The sample thinking chain includes the positive sample thinking chain; The step of selecting a sample thought chain from the multiple reasoning thought chains based on the cumulative exploration value of the terminal node of the reasoning thought chain and the explorable value of each node in the reasoning thought chain includes: From the multiple reasoning chains, select the first candidate chain whose cumulative exploration value of the termination node satisfies the first screening condition; For each node in each first candidate thought chain, calculate the average explorable value and select the first candidate thought chain with the largest average explorable value as the positive sample thought chain.

12. The method according to claim 10, characterized in that, The sample thinking chain also includes the negative sample thinking chain; The method further includes: From the multiple reasoning chains, select the second candidate reasoning chain whose cumulative exploration value of the termination node satisfies the second screening condition; For each node in each second candidate thought chain, calculate the average explorable value and select the second candidate thought chain with the smallest average explorable value as the negative sample thought chain.

13. The method according to claim 1, characterized in that, The training samples include: the query question sample and the positive sample thought chain, and the query question sample and the negative sample thought chain; The training samples are also used to fine-tune the initial process reward model to obtain the target process reward model; the target process reward model is used to assign cumulative exploration value to the obtained extended nodes during the node expansion process of problem reasoning based on the query question.

14. A method for training a reasoning model, characterized in that, The method includes: Obtain training samples constructed by any one of claims 1 to 13; Based on the training samples, a predicted query plan is obtained through an initial inference model; A first model loss is constructed based on the predicted query plan and the training samples, and the initial inference model is fine-tuned using the first model loss to obtain a target inference model; the target inference model is used to perform question inference based on the query question to obtain the corresponding query plan.

15. The method according to claim 14, characterized in that, The training samples include query question samples; The method further includes: The initial process reward model obtains the cumulative exploration value of each node in the problem reasoning process based on the query problem sample; The second model loss is constructed based on the cumulative exploration value predicted for each node and the training samples. The process reward model is then fine-tuned using the second model loss to obtain the target process reward model. The target process reward model is used to assign a cumulative exploration value to the expanded nodes during the node expansion process based on the query question.

16. A method for problem reasoning, characterized in that, The method includes: Obtain the query question, and perform node reasoning on the query question based on the target reasoning model obtained by any one of claims 14 or 15 to obtain leaf nodes; The leaf nodes are expanded to obtain extended nodes, and inference simulation is performed on the extended nodes until the terminal node is obtained; Based on the termination node, backtrack sequentially to the root node that matches the query question to obtain the query plan corresponding to the query question.

17. The method according to claim 16, characterized in that, The process of expanding the leaf node to obtain the expanded node includes: The leaf node is expanded to obtain multiple candidate expansion nodes, and the cumulative exploration value of each candidate expansion node is determined by the target process reward model. Select the candidate expansion node with the largest cumulative exploration value as the expansion node.

18. A training sample construction apparatus for a model, characterized in that, The device includes: The node reasoning module is used to perform node reasoning on the query question sample to obtain multiple leaf nodes using the initial reasoning model obtained through pre-training. The node expansion and simulation module is used to expand at least one of the leaf nodes to obtain multiple expanded nodes, and to perform inference simulation on the multiple expanded nodes until a termination node is obtained. The node backtracking module is used to backtrack to the root node that matches the query question sample based on the termination node to obtain the reasoning thought chain, and to obtain multiple reasoning thought chains by repeatedly executing node reasoning, node expansion, reasoning simulation and backtracking; The training sample construction module is used to construct training samples based on the query question samples and the multiple reasoning thought chains. The training samples are used to fine-tune the initial reasoning model to obtain a target reasoning model. The target reasoning model is used to perform question reasoning based on the query question to obtain the corresponding query plan.

19. A training device for a reasoning model, characterized in that, The device includes: A training sample acquisition module is used to acquire training samples constructed by any one of claims 1 to 13; The inference prediction module is used to obtain a predicted query plan based on the training samples and an initial inference model. The model training module is used to construct a first model loss according to the predicted query plan and the training samples, and to fine-tune the initial inference model using the first model loss to obtain a target inference model; the target inference model is used to perform question inference based on the query question to obtain the corresponding query plan.

20. A problem-solving reasoning device, characterized in that, The device includes: A node reasoning module is used to obtain the query question and perform node reasoning on the target reasoning model obtained by any one of claims 14 or 15 to obtain leaf nodes; The node expansion module is used to expand the leaf nodes to obtain expanded nodes, and to perform inference simulation on the expanded nodes until the termination node is obtained. The problem reasoning module is used to backtrack sequentially to the root node that matches the query problem based on the termination node, and obtain the query plan corresponding to the query problem.

Images