Large model agent interactive question and answer task decision-making method, device and equipment and medium

Abstract

The invention relates to a large model agent interactive question and answer task decision-making method, device and equipment and a medium. The method comprises the following steps: constructing an agent strategy model based on a partially observable Markov decision-making process; acquiring an instruction input by a user, and generating an information search track by adopting the agent strategy model; constructing a decision tree of an information search trajectory by adopting a Monte Carlo tree search algorithm to obtain a preference data set labeled by a Q value; setting a large language model, and training the large language model by adopting the step-level optimization objective function according to the preference data set marked by the Q value to obtain a trained large language model; constructing a Q value model according to the trained large language model, and then outputting Q value estimation according to the Q value model; and according to Q value estimation and information search trajectory selection actions, generating a question and answer process. According to the method, the expression and generalization ability of the large model agent in the autonomous reasoning and decision-making process can be remarkably enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium. Background Art

[0002] Interactive question-answering systems are a type of artificial intelligence application that directly communicates with users through natural language understanding and generation technology. Its goal is to answer users' questions in real time or help users complete complex tasks in multiple rounds of dialogue. In recent years, question-answering systems based on large models (such as GPT-4) have made significant progress in dialogue generation and language understanding, and can handle more natural and complex dialogue content.

[0003] Although these models have shown great potential, there are still many challenges in handling complex multi-turn dialogue tasks. In interactive question-answering scenarios, feedback on tasks is usually sparse in most cases, and success or failure is only clearly indicated at the end of the dialogue. As the complexity of the dialogue increases, the length and complexity of planning the dialogue flow also increase accordingly, which may cause the agent to accumulate errors and uncertainties over multiple dialogue turns.

[0004] Traditional interactive question-answering systems usually use the Q function in reinforcement learning to guide decision-making. The Q function can measure the expected effect of choosing a response in the current dialogue state. However, large model-driven dialogue systems often rely on greedy decoding strategies, in which responses are obtained by sampling from the language model. Therefore, the selected response is not necessarily the best choice in the dialogue, and choosing the wrong response often makes the dialogue goal unattainable. In addition, the lack of support for the Q value model also makes these systems lack effective feedback distribution capabilities during the dialogue process, making it difficult to clearly understand the importance of each response to the final goal.

[0005] To solve these problems, existing research attempts to use tree-based search methods to solve the above problems. However, such methods usually build a search tree before reasoning about the decision, which not only brings additional computational overhead, but also requires the assumption that the environment can be backtracked, which limits the application of existing methods. Summary of the invention

[0006] Based on this, it is necessary to provide a large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium that does not require the assumption of an environment and can be traced back, and can enhance the performance and generalization ability of the large-model intelligent agent in the process of autonomous reasoning and decision-making.

[0007] A large-model intelligent agent interactive question-answering task decision-making method, the method comprising:

[0008] Modeling agent strategies based on partially observable Markov decision processes;

[0009] Obtaining user input instructions, using the agent strategy model to generate an information search trajectory; using the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectory, and obtaining a preference data set labeled with a Q value;

[0010] Setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model;

[0011] Building a Q-value model based on the trained large language model, and then outputting a Q-value estimate based on the Q-value model;

[0012] An action is selected according to the Q-value estimation and the information search trajectory to generate a question-answering process.

[0013] A large-model intelligent agent interactive question-answering task decision-making device, the device comprising:

[0014] An agent strategy model building module, which is used to build an agent strategy model based on a partially observable Markov decision process;

[0015] A preference data set generation module is used to obtain user input instructions, use the agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to build a decision tree of the information search trajectory, and obtain a preference data set labeled with a Q value;

[0016] A Q-value model building module is used to set a large language model for learning Q-values, and train the large language model using a step-level optimization objective function according to the preference data set labeled with the Q-values ​​to obtain a trained large language model;

[0017] A Q-value estimation calculation module is used to construct a Q-value model according to the trained large language model, and then output a Q-value estimate according to the Q-value model;

[0018] The question-answering process generation module selects actions according to the Q-value estimation and the information search trajectory to generate a question-answering process.

[0019] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the large-model intelligent agent interactive question-answering task decision method when executing the computer program.

[0020] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the large-model intelligent agent interactive question-answering task decision-making method.

[0021] The above-mentioned large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium first build an intelligent agent strategy model based on a partially observable Markov decision process; then obtain user input instructions and use the intelligent agent strategy model to generate an information search trajectory; use the Monte Carlo tree search algorithm to build a decision tree of the information search trajectory to obtain a preference data set labeled with Q values; in addition, set up a large language model for learning Q values, and train the large language model according to the preference data set labeled with Q values ​​using a step-level optimization objective function to obtain a trained large language model; build a Q value model based on the trained large language model, and then output a Q value estimate based on the Q value model; select actions based on the Q value estimate and the information search trajectory to generate a question-answering process.

[0022] The present invention combines a large model agent with a Monte Carlo tree search algorithm, and can obtain a high-quality Q-value labeled preference data set without assuming that the environment can be traced back and without manual annotation, thereby effectively supporting the exploration of high-quality question-answering interaction trajectories; at the same time, by additionally setting up a large language model, and using the Q-value labeled preference data set and the step-level optimization objective function to train the large language model, it can well refine the strategic improvement of the information search trajectory; then the Q-value model is constructed by the trained large language model, which can significantly improve the performance of the large model agent, so that it can also have a good performance in complex tasks. In addition, the present invention does not need to fine-tune the backbone of the large model of the agent, but evaluates each step of the trajectory through the optimized Q-value model, and then selects the action accordingly, which significantly enhances the performance and generalization ability of the large model agent in the autonomous reasoning and decision-making process. In addition, since there is no need to fine-tune the backbone of the large model of the agent, the method proposed by the present invention can be efficiently combined with other large models, showing good compatibility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0024] Figure 1 A flowchart of a large model agent interactive question-answering task decision method in one embodiment;

[0025] Figure 2 A schematic diagram of constructing a preference data set annotated with Q values ​​in one embodiment;

[0026] Figure 3A schematic diagram of the effect of the amount of preference data at the step level on model performance in one embodiment;

[0027] Figure 4 A schematic diagram of the result of constructing preference data with different Monte Carlo tree search algorithm iteration times in one embodiment;

[0028] Figure 5 FIG. 1 is a schematic diagram of the result of evaluating the optimized Q value model in one embodiment, wherein Figure 5 (a) is a schematic diagram of the Q-value model preference accuracy. Figure 5 (b) Schematic diagram of the Q-value density distribution of 200 actions sampled from the WebShop task test set, which leads to success and failure;

[0029] Figure 6 It is a structural block diagram of a large model intelligent agent interactive question-answering task decision-making device in one embodiment;

[0030] Figure 7 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment.

[0031] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0033] It should be noted that in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0034] The following will describe the implementation of the present invention in detail with reference to the accompanying drawings in the embodiment diagram of the present invention.

[0035] In one embodiment, Figure 1 As shown, a large-model intelligent agent interactive question-answering task decision method is provided, including the following steps:

[0036] Step 201, constructing an agent strategy model based on a partially observable Markov decision process.

[0037] It can be understood that the partially observable Markov decision process is a framework for modeling decision problems, which is applicable to complex environments where the state is not completely visible or certain. In actual interactive question-answering tasks, the agent can only make inferences based on past observations and historical decisions. Therefore, building an agent strategy model through a partially observable Markov decision process can effectively model this uncertainty and partial observability, allowing the agent to make effective decisions and plans in the absence of complete information.

[0038] Step 202, obtain user input instructions, use the intelligent agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to build a decision tree of the information search trajectory, and obtain a preference data set labeled with Q values.

[0039] It can be understood that in a given environment, the user inputs an instruction, and the agent strategy model generates an action under the instruction (including an inference result and an executable prompt). The agent then performs state transition according to the action, integrates relevant information resources, continues to interact with the user, and generates information search trajectories until the task is completed or the maximum time step is reached.

[0040] At the same time, when generating information search trajectories, a decision tree is constructed through the Monte Carlo tree search algorithm for the selection of actions in the decision-making process of the large model agent. The Monte Carlo tree search algorithm is a heuristic search algorithm that is mainly used for optimization problems in the decision-making process. Through simulation and optimization, it can efficiently find the optimal action in the question-answering process. By combining the agent strategy model with the Monte Carlo tree search algorithm and using the node values ​​of the decision tree as supervised preference data, a high-quality Q-value labeled preference data set can be generated, which provides a basis for subsequent large language model training.

[0041] Step 203, setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model.

[0042] Step 204, construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model.

[0043] Step 205, selecting actions based on the Q-value estimation and the information search trajectory to generate a question-answering process.

[0044] It is understandable that traditional methods for enhancing the capabilities of large model agents, such as prompt-based methods, usually use prompt-based mechanisms to improve the capabilities of agents. This method relies on manual intervention annotations. However, as the complexity of the task increases, the difficulty of designing appropriate prompts also increases. Fine-tuning-based methods generally use rejection sampling fine-tuning, reinforcement learning (RL) or its variants to fine-tune large models by using trajectories with result feedback. In the fine-tuning process, instability is very likely to occur, generalization ability is poor, and a large amount of computing resources are required, and the calculation process is complex.

[0045] Based on this, the present invention sets up an additional large language model for learning Q-values, and trains it through a preference data set annotated with Q-values, without manual annotation. Then, a Q-value model is constructed based on the trained large language model, so that the Q-value model has better generalization ability and can accurately evaluate the value of each action in the entire trajectory process, thereby improving the decision-making ability and task execution effect of the large model intelligent body, so that it can complete reasoning and decision-making tasks more accurately and efficiently in complex environments.

[0046] For the Q-value model built by the trained large language model, the large model agent can generate the action with the highest Q-value that interacts with the user. By sampling multiple candidate actions in the decision tree expansion stage, and then selecting actions based on the Q-value interaction with the user, the Q-value estimation generates the result corresponding to the user input instruction, and finally generates the question-answering process.

[0047] In the above-mentioned large-model intelligent agent interactive question-answering task decision-making method, first, an intelligent agent strategy model is constructed based on a partially observable Markov decision process; then, user input instructions are obtained, and the intelligent agent strategy model is used to generate an information search trajectory; a decision tree of the information search trajectory is constructed using a Monte Carlo tree search algorithm to obtain a preference data set labeled with a Q value; in addition, a large language model for learning Q values ​​is set up, and the large language model is trained using a step-level optimization objective function based on the preference data set labeled with a Q value to obtain a trained large language model; a Q value model is constructed based on the trained large language model, and then a Q value estimate is output based on the Q value model; actions are selected based on the Q value estimate and the information search trajectory to generate a question-answering process.

[0048] The present invention combines a large model agent with a Monte Carlo tree search algorithm, and can obtain a high-quality Q-value labeled preference data set without assuming that the environment can be traced back and without manual annotation, thereby effectively supporting the exploration of high-quality question-answering interaction trajectories; at the same time, by additionally setting up a large language model, and using the Q-value labeled preference data set and step-level optimization objective function to train the large language model, it can well refine the strategic improvement of the information search trajectory; then the Q-value model is constructed by the trained large language model, which can significantly improve the performance of the large model agent, so that it can also have a good performance in complex tasks. In addition, the present invention does not need to fine-tune the backbone of the large model of the agent, but constructs a Q-value model by the trained large language model to evaluate each step of the trajectory, and then selects the action accordingly, which significantly enhances the performance and generalization ability of the large model agent in the autonomous reasoning and decision-making process. In addition, since there is no need to fine-tune the backbone of the large model of the agent, the method proposed by the present invention can be efficiently combined with other large models, showing good compatibility and adaptability.

[0049] In one embodiment, an agent strategy model is constructed based on a partially observable Markov decision process, including:

[0050] According to the given environment, the agent interactive question-answering task is converted into a partially observable Markov decision process, which can be expressed as:

[0051] The agent strategy model is constructed based on the partially observable Markov decision process, which is expressed as:

[0052]

[0053] In the formula, Represents all possible user input instructions, that is, all types of queries that the agent needs to handle, such as user queries or questions; Represents all possible states encountered during the question-answering process, such as the agent’s understanding of the question, known information, retrieved documents, and the current derived answer state; Represents the set of all possible actions that the agent can take in each state, such as asking questions, retrieving information, integrating information, reasoning, or generating part of an answer; Represents all the information resources available to the agent, including user feedback, retrieved document fragments, and related information extraction results; represents the probability of updating the state of the question understanding or answer after the agent performs an action (such as retrieval or reasoning); R represents the reward function, which is a binary reward based on the correctness of the answer (correct or wrong); π represents the agent strategy function; u represents the user input command in a given environment; τ represents the information search trajectory; a trepresents action; t represents the current time step; T represents the total number of action steps; τ t-1 represents the trajectory of the previous time step.

[0054] In one embodiment, obtaining a user input instruction and using an agent strategy model to generate an information search trajectory includes:

[0055] Get the user input command u in a given environment and generate the information search trajectory at the current time step t. The information search trajectory expression is:

[0056] τ t =(a0,o0,...,a t ,o t )~π(τ t |u);

[0057] In the formula, o t Represents the information resources obtained by the agent at the current time step t.

[0058] It can be understood that in a partially observable Markov decision process In the function Expressed as The function R is expressed as

[0059] Specifically, for a user input command u in a given environment, the large model agent generates an action a1~π(·|u) according to its strategy π, and the agent state is transferred to The agent observes Then continue to interact with the user until the task is completed or the maximum time step is reached. At time step t, given the history and current observation, the agent generates a subsequent action a t+1 ~π(·|u,a0,o0,...,a t ,o t ). Based on this, we construct an agent strategy model for multi-step interactive question-answering tasks and generate information search trajectories at time step t. It is worth noting that action a t Included is a reasoning thought and an actionable tip.

[0060] In one embodiment, a decision tree of information search trajectories is constructed using a Monte Carlo tree search algorithm to obtain a preference data set labeled with a Q value, including:

[0061] Define a decision tree where nodes represent states, edges represent actions, and store a set of statistics at each node;

[0062] Start expanding from the root node. At each depth, calculate the upper confidence bound based on the statistical data stored in each node to select the appropriate leaf node for the next trajectory expansion.

[0063] When performing the next step of trajectory expansion, sample several actions a from the information search trajectory t Expand the leaf nodes until the maximum number of action steps is reached to generate a decision tree;

[0064] Obtain the preference data set with Q value annotation according to the decision tree.

[0065] In one of the embodiments, when generating a decision tree, it also includes: evaluating the expanded nodes through a rollout algorithm to generate an evaluation result; and back-propagating and updating the node data of the decision tree according to the evaluation result to obtain a decision tree.

[0066] Specifically, when generating information search trajectories, each planning path consists of several steps, so it is of great significance to obtain the value estimation of each intermediate step. The present invention uses the Monte Carlo tree search algorithm to construct a decision tree, and defines that each node in the decision tree represents a state s, and each edge represents an action a t Since in partially observable Markov decision processes, the complete state is not available at every step Only some of the observed Therefore, for the decision tree, the root node s0 contains (u), and the other nodes use the currently observed In addition, each node stores a set of statistics:

[0067]

[0068] Where V(s) represents the value function (expected return) of the subtree from node s, Indicates the number of visits to node s.

[0069] After determining the framework of the decision tree, we start to expand the tree. First, we select the trajectory from the root node to the leaf node. At each depth of the decision tree, we calculate the upper confidence bound value based on the statistical data stored in each node to select the appropriate leaf node for the next trajectory expansion. The expression of the upper confidence bound is:

[0070]

[0071] Where w represents the exploration weight, and p(s) is the parent node of node s.

[0072] The upper confidence bound can provide a good balance between exploration and exploitation.

[0073] Then, by sampling n actions a from the information search trajectory tThe leaf nodes are expanded. It can be understood that, unlike the action space of traditional agents, which is usually limited, the action space of large model agents is infinite. Therefore, large models can generate an infinite number of unique actions (token sequences), even if they are invalid. Based on this, the present invention uses high temperature sampling of multiple actions to ensure diversity, and the environment processes each action a t , and provide corresponding feedback as observation, generate n new leaf nodes as extensions, and incorporate them into the tree.

[0074] However, the decision tree constructed in this way is shallow, and the expansion process will quickly reach the terminal node. Therefore, the expanded nodes are evaluated through the rollout algorithm. Specifically, starting from the expanded leaf node, the large model agent interacts with the user until the interaction ends or the maximum rollout depth is reached. If the explored node achieves the goal, the result provided is a reward; otherwise, the exploration node with the maximum depth is set to a fixed negative reward.

[0075] After the evaluation is completed, the node data of the tree is updated through back propagation. The update is performed based on the evaluation results of the evaluation phase, that is, the result is a reward or a fixed negative reward. t For each node in, N(s) increases by 1, and the value is from the terminal node s T Update to the root node s0, the expression of back propagation update is:

[0076]

[0077] The updated value is used in the UCT formula to guide the selection of the next node.

[0078] Finally, after multiple selection, expansion, evaluation, and back-propagation iterations, a decision tree is obtained. Specifically, according to the node reward r(s), the best leaf node for the agent to explore is first found, and then the corresponding trajectory is obtained using a top-down approach. At each layer of depth, a part of the best trajectory is taken as the unfinished trajectory segment τ T . Then the preferred action for the next step is Taken from the best trajectory, with less preferred actions Select from the candidate actions with the smallest value, the preference data marked by Q value is as follows Figure 2 As shown in Figure 2. The white nodes represent the best trajectories explored by the agent and are considered as the selected nodes at each depth of the tree. The gray nodes are candidate nodes for building less preferred samples, while the black nodes are ignored.

[0079] It can be understood that the preference data marked with Q values ​​refers to the preference of the data in the data set reflected by the Q value. By continuously training the large language model through repeated cycles and further updating the Q value model, it is equivalent to continuously training the Q value model through the preference data marked with Q values. Therefore, the actions in the entire information search trajectory are affected by the Q value. Actions with higher Q value estimates will be more "preferred" and become the optimal actions, which in turn helps the large model agent generate a question-and-answer process that is more in line with the specific environment.

[0080] By training a large language model using step-level preference data, the constructed Q-value model can have a significant improvement in sample efficiency. Figure 3 , shows the impact of the number of step-level preference data on model performance, which evaluates the effect of different numbers of training samples on the performance of the Q-value model on the HotPotQA task. In evaluating the HotPotQA task, several checkpoints of a cycle of training a large language model and updating the Q-value model are used, representing different numbers of training samples. It can be seen that less than 400 step-level preference data points can significantly improve performance, and only about 250 task instructions are required to implement it. This proves that the Q-value model constructed based on the trained large language model by training the large language model with step-level preference data can significantly improve sample efficiency. Therefore, the efficiency and effect of the constructed Q-value model can be understood through different numbers of step-level preference data, so as to select the appropriate number.

[0081] See also Figure 4 , shows the results of constructing a Q-value labeled preference dataset using different numbers of Monte Carlo tree search algorithm iterations. It analyzes the impact of increasing the number of Monte Carlo tree search algorithm iterations on the number of successful trajectories collected and the number of step-level preference data points. Figure 4 It is possible to understand the impact of setting the number of Monte Carlo Tree Search algorithm iterations during the data collection process on the size and quality of the final dataset. Specifically, the number of successful trajectories used to construct step-level preference data increases with the number of Monte Carlo Tree Search algorithm iterations. Almost all Monte Carlo Tree Search algorithm processes terminate early before reaching the maximum reward or depth, usually before the 50th iteration, so additional iterations are redundant. In addition, as the number of Monte Carlo Tree Search algorithm iterations increases, the number of step-level preference data points increases faster than the number of successful trajectories. This is because trajectories explored by more Monte Carlo Tree Search algorithm iterations usually involve more decision steps, thus providing more step-level preference data.

[0082] In one embodiment, according to the preference data set annotated with Q value, a large language model is trained using a step-level optimization objective function to obtain a trained large language model, including:

[0083] Preference dataset labeled by Q value Get preference data pair

[0084] Then the expression of the step-level optimization objective function is:

[0085]

[0086] In the formula, β represents the weight of the constraint; π Q Represents a large language model; represents the reference model; Actions that indicate best preferences; represents the less preferred action; u represents the user input command in a given environment; τ t represents the information search trajectory at the current time step t; represents the averaging operation; σ represents the activation function.

[0087] In one embodiment, the expression of the Q value model is:

[0088]

[0089] It can be understood that, unlike the traditional direct policy optimization (DPO) which targets the trajectory level, the present invention uses a more fine-grained Q-value annotated preference data set. By building a step-level optimization objective function, we can achieve direct policy optimization (DPO) at the step level, which can provide more detailed and precise results by observing the specific steps more deeply, making the trained large language model π Q It has better generalization performance, which makes the Q-value model have better generalization performance. In addition, by constructing the Q-value model by additionally training a large language model instead of fine-tuning the large model backbone of the intelligent agent, the method proposed in the present invention can be integrated with both open source and API-based large model intelligent agents.

[0090] In one of the embodiments, the superiority of the proposed method is verified. The proposed method is annotated with (Ours), and in the interactive question-answering scenario, the dataset uses HotPotQA. HotPotQA is a question answering task that requires searching in Wikipedia articles. The large model agent is equipped with API calls for searching and retrieving information. After receiving the answer, the environment provides a binary reward of 0 or 1 based on its correctness. The maximum number of steps is set to 7. When the agent chooses the "buy" action, the environment provides a final reward from 0 to 1 based on the matching heuristic of product attributes and prices. The maximum number of steps is set to 10. All methods in the experiment use the ReAct-style (Yao et al., 2023b) interaction format, generate CoT (Wei et al., 2022) thinking before the action, the maximum context length is set to 4096, and the number of candidate actions is set to n=5.

[0091] In addition, the effectiveness of the proposed method is also verified in an automatic shopping scenario. The WebShop task requires the agent to solve the shopping task by browsing a website with detailed product descriptions and specifications. The available action APIs include SEARCH[QUERY] for using the search bar and CHOOSE[BUTTON] for clicking buttons on the web page. Clickable buttons include product title, options, buy, return to search, and previous / next page. When the user selects the "buy" action, the environment provides a final reward from 0 to 1 based on the matching heuristic of product attributes and prices.

[0092] Training settings: Before collecting step-level preference data, the maximum number of iterations of Monte Carlo tree search is set to 30. When building the Q-value model, Phi-1.5 is mainly used for large language models. In the HotPotQA task, 1000 questions are randomly selected for training, 100 questions are used for validation, and 100 questions are used for testing. In the WebShop task, the data split of Song et al. (2024) is followed, including 1824 instructions for training and 200 instructions for testing. The batch size is set to 16, the learning rate is set to 1e-5, the warm-up ratio is 0.1, the cosine scheduler is used, and the β in the DPO loss is set to 0.1. All experiments (except for the fine-tuning-based comparison method) are performed on the NVIDIAA4048G GPU. The experimental results are shown in Table 1.

[0093]

[0094] It can be seen that the Q-value model proposed in the present invention can significantly enhance decision-making capabilities. The Q-value model proposed in the present invention performs twice as well as the large model agent established by Phi-3-mini-4k-instruct in the WebShop task, and improves performance by 75% in the HotPotQA task. Since BoN is a strong baseline, it has the same computational overhead as the method proposed in the present invention, but has additional result rewards from the environment. However, the method proposed in the present invention outperforms BoN in both tasks. Compared with the most advanced commercial large model agent based on API, the method proposed in the present invention outperforms the lightweight API-based GPT-4o-mini in both tasks, and even outperforms the most effective GPT-4-turbo in the WebShop task.

[0095] Building a Q-value model by training a large language model is more efficient and effective than fine-tuning a large language model (LLM) backbone. Using demonstrated trajectories to supervise the RFT of fine-tuning the LLM can improve the performance of the agent task. In addition, AgentEval using more reward information can further improve the performance of the WebShop task, but not the HotPotQA task. This is because the HotPotQA environment only provides binary rewards. ETO adds more negative samples for learning and achieves the best results among the fine-tuning-based methods. This highlights the necessity of incorporating imperfect trajectories into training. Fine-tuning the LLM backbone requires high-performance computing resources, especially when the LLM size and context length are large. Therefore, the comparison of the proposed method with the fine-tuning-based method is mainly based on the Phi-3-mini-4k-instruct with 3.8B parameters. In contrast, the Q-value model proposed in the present invention is built by the lighter Phi-1.5 with 1.3B parameters. Nevertheless, the proposed method is more effective than all the above-mentioned fine-tuning-based methods.

[0096] The Q-value model constructed by the present invention can be generalized to different LLM backbones. The Q-value function simulates the future total reward in the environment. Therefore, it can be generalized to different large-model agents within the task. To verify this, the preference data sampled by Phi-3-mini-4k-instruct was first used to train a large language model and build a Q-value model. Then, the constructed Q-value model was directly applied to more powerful open source large-model Llama-3.1-8B-instruct and API-based LLMs, including GPT-4o-mini and GPT-4-turbo. Even if the performance improvement is not as high as when applied to a large-model agent with sampled training data, decision-making ability can continue to improve. This is because the actions sampled by the more powerful large-model agent can be regarded as out-of-distribution (OOD) relative to the step-level preference data collected by Phi-3-mini-4k-instruct for training a large language model and then building a Q-value model. This is a promising way to use the trial-and-error experience of weak and cheap large-model agents to assist more powerful or API-based expensive large-model agents.

[0097] Furthermore, the accuracy of the Q-value model constructed by the present invention in evaluating the preference relationship of the collected step-level data is also studied. Figure 5 As shown in (a), it can be observed that the preference relations in the training set can be easily learned in both tasks. The accuracy of the IND and OOD test sets decreases successively, which is about 80% in the WebShop task and about 65% in the HotPotQA task. As mentioned above, the result reward provided by the HotPotQA environment is binary. In addition to early stopping when reward 1 is obtained, as shown in Figure 5 As shown in (b), the Q-values ​​of the nodes are as distinct as those in the WebShop task. This would suggest designing more complex rewards to further improve performance. Nevertheless, this preference accuracy is sufficient to improve performance on downstream tasks.

[0098] To further evaluate the effectiveness of the Q-value model constructed by the present invention, the present invention randomly selected 200 actions from the successful and failed trajectories, respectively, and visualized their Q-values ​​in the figure. All trajectories were sampled from the test instructions of the WebShop task by Phi-3-mini-4k-instruct. The Q-value distribution of actions in the failed trajectories is skewed to the left. In contrast, the distribution of incorrect solutions shows a smaller inclination, with most probability densities skewed to the right. This pattern may be because failures usually stem from choosing some destructive actions, which shows that the Q-value model constructed by the present invention has the ability to assign credit.

[0099] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0100] In one embodiment, Figure 6 As shown, a large-model intelligent agent interactive question-answering task decision-making device is provided, including: an intelligent agent strategy model building module 401, a preference data set generation module 402, a large language model training module 403, a Q-value estimation calculation module 404 and a question-answering process generation module 405, wherein:

[0101] The intelligent agent strategy model building module 401 is used to build an intelligent agent strategy model based on a partially observable Markov decision process.

[0102] The preference data set generation module 402 is used to obtain user input instructions, generate information search trajectories using the intelligent agent strategy model, and construct a decision tree of the information search trajectory using the Monte Carlo tree search algorithm to obtain a preference data set labeled with a Q value.

[0103] The large language model training module 403 is used to set a large language model for learning Q values, and train the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model.

[0104] The Q-value estimation calculation module 404 is used to construct a Q-value model according to the trained large language model, and then output a Q-value estimate according to the Q-value model.

[0105] The question-answering process generating module 405 is used to select actions according to the Q-value estimation and the information search trajectory to generate the question-answering process.

[0106] For the specific limitations of the large-model intelligent agent interactive question-answering task decision-making device, please refer to the limitations of the large-model intelligent agent interactive question-answering task decision-making method above, which will not be repeated here. Each module in the above-mentioned large-model intelligent agent interactive question-answering task decision-making device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store large-model intelligent agent interactive question-answering task decision data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large-model intelligent agent interactive question-answering task decision method is implemented.

[0108] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present invention, and does not constitute a limitation on the computer device to which the scheme of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0110] Step 201, constructing an agent strategy model based on a partially observable Markov decision process.

[0111] Step 202, obtain user input instructions, use the intelligent agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to build a decision tree of the information search trajectory, and obtain a preference data set labeled with Q values.

[0112] Step 203, setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model.

[0113] Step 204, construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model.

[0114] Step 205, selecting actions based on the Q-value estimation and the information search trajectory to generate a question-answering process.

[0115] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0116] Step 201, constructing an agent strategy model based on a partially observable Markov decision process.

[0117] Step 202, obtain user input instructions, use the intelligent agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to build a decision tree of the information search trajectory, and obtain a preference data set labeled with Q values.

[0118] Step 203, setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model.

[0119] Step 204: construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model. Step 205: select an action based on the Q-value estimate and the information search trajectory to generate a question-answering process.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the invention. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A large-model intelligent agent interactive question-answering task decision-making method, characterized in that: The method comprises: Modeling agent strategies based on partially observable Markov decision processes; Obtaining user input instructions, using the agent strategy model to generate an information search trajectory; using the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectory, and obtaining a preference data set labeled with a Q value; Setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model; Building a Q-value model based on the trained large language model, and then outputting a Q-value estimate based on the Q-value model; An action is selected according to the Q-value estimation and the information search trajectory to generate a question-answering process.

2. The large-model agent interactive question-answering task decision-making method according to claim 1 is characterized in that: Build an agent strategy model based on a partially observable Markov decision process, including: According to the given environment, the agent interactive question-answering task is converted into a partially observable Markov decision process, which can be expressed as: The agent strategy model is constructed according to the partially observable Markov decision process, which is expressed as: In the formula, Represents all possible user input commands; Represents all possible states that may be encountered during the question-answering process; represents the set of all possible actions that the agent can take in each state; Represents all the information resources available to the agent; represents the probability of updating the state of the question understanding or answer after the agent performs an action; R represents the reward function; π represents the agent strategy function; u represents the user input command in a given environment; τ represents the information search trajectory; a t represents action; t represents the current time step; T represents the total number of action steps; τ t-1 represents the trajectory of the previous time step.

3. The large-model intelligent agent interactive question-answering task decision-making method according to claim 2 is characterized in that: Obtaining a user input instruction and using the agent strategy model to generate an information search trajectory includes: Get the user input command u in a given environment and generate the information search trajectory at the current time step t. The information search trajectory expression is: t t =(a0,o0,...,a t ,o t )~π(τ t |u); In the formula, o t Represents the information resources obtained by the agent at the current time step t.

4. The large-model intelligent agent interactive question-answering task decision-making method according to any one of claims 1 to 3, characterized in that: A decision tree of the information search trajectory is constructed using a Monte Carlo tree search algorithm to obtain a preference data set labeled with Q values, including: Define a decision tree where nodes represent states, edges represent actions, and store a set of statistics at each node; Start expanding from the root node. At each depth, calculate the upper confidence bound based on the statistical data stored in each node to select the appropriate leaf node for the next trajectory expansion. When performing the next step of trajectory expansion, a number of actions a are sampled from the information search trajectory. t Expand the leaf nodes until the maximum number of action steps is reached to generate a decision tree; A preference data set labeled with Q values ​​is obtained according to the decision tree.

5. The large-model intelligent agent interactive question-answering task decision-making method according to claim 4 is characterized in that: When generating a decision tree, it also includes: Evaluate the expanded nodes through the rollout algorithm and generate evaluation results; The node data of the decision tree is updated by back propagation according to the evaluation result to obtain a decision tree.

6. The large-model intelligent agent interactive question-answering task decision-making method according to claim 4 is characterized in that: According to the preference data set annotated with the Q value, the large language model is trained using a step-level optimization objective function to obtain a trained large language model, including: Preference dataset annotated according to the Q value Get preference data pair Then the expression of the step-level optimization objective function is: In the formula, β represents the weight of the constraint; π Q Represents a large language model; represents the reference model; Actions that indicate best preferences; represents the less preferred action; u represents the user input command in a given environment; τ t represents the information search trajectory at the current time step τ; represents the averaging operation; σ represents the activation function.

7. The large-model agent interactive question-answering task decision method according to claim 6 is characterized in that: The expression of the Q value model is:

8. A large-model intelligent agent interactive question-answering task decision-making device, characterized in that: The device comprises: An agent strategy model building module, which is used to build an agent strategy model based on a partially observable Markov decision process; A preference data set generation module is used to obtain user input instructions, use the agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to build a decision tree of the information search trajectory, and obtain a preference data set labeled with a Q value; A large language model training module is used to set a large language model for learning Q values, and train the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model; A Q-value estimation calculation module is used to construct a Q-value model according to the trained large language model, and then output a Q-value estimate according to the Q-value model; A question-answering process generation module is used to select actions according to the Q value estimation and the information search trajectory to generate a question-answering process.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, it implements the steps of the large-model intelligent agent interactive question-answering task decision method described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large-model intelligent agent interactive question-answering task decision method described in any one of claims 1 to 7 are implemented.

Images