Large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium

Abstract

The present invention relates to a large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium, the method comprising: constructing an intelligent agent strategy model based on a partially observable Markov decision process; obtaining user input instructions, and using the intelligent agent strategy model to generate an information search trajectory; using a Monte Carlo tree search algorithm to construct a decision tree of the information search trajectory to obtain a preference data set labeled with Q values; setting a large language model, and training the large language model using a step-level optimization objective function based on the preference data set labeled with Q values ​​to obtain a trained large language model; constructing a Q-value model based on the trained large language model, and then outputting a Q-value estimate based on the Q-value model; selecting an action based on the Q-value estimate and the information search trajectory to generate a question-answering process. The present invention can significantly enhance the performance and generalization ability of large-model intelligent agents in autonomous reasoning and decision-making processes.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium. Background Art

[0002] Interactive question-answering systems are a type of AI application that uses natural language understanding and generation techniques to engage in direct conversations with users. Their goal is to answer user questions in real time or help users complete complex tasks in multi-turn conversations. In recent years, question-answering systems based on large models (such as GPT-4) have made significant progress in dialogue generation and language understanding, enabling them to handle more natural and complex conversations.

[0003] While these models have shown great potential, numerous challenges remain when tackling complex, multi-turn dialogue tasks. In interactive question-answering scenarios, feedback on tasks is often sparse, with success or failure only clearly indicated at the end of the conversation. As the complexity of the conversation increases, the length and complexity of planning the dialogue flow also increase, potentially leading to the accumulation of errors and uncertainty across multiple turns.

[0004] Traditional interactive question-answering systems typically use the Q-function from reinforcement learning to guide decision-making. The Q-function measures the expected effect of selecting a response in the current dialogue state. However, large-model-driven dialogue systems often rely on greedy decoding strategies, in which responses are sampled from the language model. As a result, the selected response is not necessarily the optimal one for the conversation, and choosing the wrong response often prevents the dialogue goal from being achieved. Furthermore, the lack of support for Q-value models also hinders these systems' ability to effectively allocate feedback during the dialogue process, making it difficult to clearly understand the importance of each response to the ultimate goal.

[0005] To address these issues, existing research has attempted to use tree-based search methods. However, these methods typically construct a search tree before reasoning about a decision, which not only incurs additional computational overhead but also requires the assumption that the environment can be backtracked, limiting the applicability of existing methods. Summary of the Invention

[0006] Based on this, it is necessary to provide a large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium that does not require an assumed environment and can be traced back, and can enhance the performance and generalization ability of large-model intelligent agents in autonomous reasoning and decision-making processes, in order to address the above technical problems.

[0007] A large-model intelligent agent interactive question-answering task decision-making method, the method comprising:

[0008] Build agent strategy models based on partially observable Markov decision processes;

[0009] Obtaining user input instructions, using the agent strategy model to generate an information search trajectory; using the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectory, and obtaining a preference data set annotated with Q values;

[0010] Setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model;

[0011] Build a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model;

[0012] An action is selected based on the Q-value estimation and the information search trajectory to generate a question-answering process.

[0013] A large-model intelligent agent interactive question-answering task decision-making device, comprising:

[0014] An agent policy model building module, which is used to build an agent policy model based on a partially observable Markov decision process;

[0015] A preference data set generation module is used to obtain user input instructions and use the agent strategy model to generate an information search trajectory; a Monte Carlo tree search algorithm is used to construct a decision tree of the information search trajectory to obtain a preference data set annotated with a Q value;

[0016] A Q-value model construction module is used to set up a large language model for learning Q-values, and train the large language model using a step-level optimization objective function based on the preference data set labeled with the Q-values ​​to obtain a trained large language model;

[0017] A Q-value estimation calculation module is used to construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model;

[0018] The question-answering process generation module selects actions according to the Q-value estimation and the information search trajectory to generate a question-answering process.

[0019] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the large-model intelligent agent interactive question-answering task decision-making method are implemented.

[0020] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the large-model intelligent agent interactive question-answering task decision-making method.

[0021] The above-mentioned large-model intelligent agent interactive question-answering task decision-making method, device, equipment and medium first construct an intelligent agent strategy model based on a partially observable Markov decision process; then obtain user input instructions and use the intelligent agent strategy model to generate an information search trajectory; use the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectory to obtain a preference data set labeled with Q values; in addition, set up a large language model for learning Q values, and use the step-level optimization objective function to train the large language model based on the preference data set labeled with Q values ​​to obtain a trained large language model; construct a Q value model based on the trained large language model, and then output a Q value estimate based on the Q value model; select actions based on the Q value estimate and the information search trajectory to generate a question-answering process.

[0022] The present invention combines a large-model intelligent agent with a Monte Carlo tree search algorithm. Without the need to assume that the environment is traceable and without manual annotation, a high-quality Q-value labeled preference data set can be obtained, thereby effectively supporting the exploration of high-quality question-answering interaction trajectories. At the same time, by adding a large language model and using the Q-value labeled preference data set and step-level optimization objective function to train the large language model, it is possible to well refine the strategic improvement of the information search trajectory. Then, the Q-value model is constructed by the trained large language model, which can significantly improve the performance of the large-model intelligent agent, so that it can also perform well in complex tasks. In addition, the present invention does not need to fine-tune the backbone of the intelligent agent's large model. Instead, it evaluates each step of the trajectory through the optimized Q-value model and then selects the action accordingly, which significantly enhances the performance and generalization ability of the large-model intelligent agent in the autonomous reasoning and decision-making process. In addition, since there is no need to fine-tune the backbone of the intelligent agent's large model, the method proposed by the present invention can be efficiently combined with other large models, showing good compatibility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0024] Figure 1 1. A flowchart of a large-model intelligent agent interactive question-answering task decision-making method in one embodiment;

[0025] Figure 2 A schematic diagram of constructing a preference dataset labeled with Q values ​​in one embodiment;

[0026] Figure 3A schematic diagram illustrating the effect of the amount of step-level preference data on model performance in one embodiment;

[0027] Figure 4 A schematic diagram of the results of constructing preference data for different Monte Carlo tree search algorithm iteration times in one embodiment;

[0028] Figure 5 Schematic diagram of the evaluation results of the optimized Q value model in one embodiment, wherein Figure 5 (a) is a diagram showing the accuracy of the Q-value model preference. Figure 5 (b) Schematic diagram of the Q-value density distribution of actions that lead to success and failure, among 200 actions sampled from the WebShop task test set;

[0029] Figure 6 This is a structural block diagram of a large-model intelligent agent interactive question-answering task decision-making device in one embodiment;

[0030] Figure 7 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment.

[0031] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0033] It should be noted that the terms "first," "second," and so on, used in this disclosure are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this disclosure, "plurality" means at least two, such as two or three, unless otherwise specifically defined.

[0034] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings.

[0035] In one embodiment, Figure 1 As shown, a large-model intelligent agent interactive question-answering task decision method is provided, which includes the following steps:

[0036] Step 201: construct an agent strategy model based on a partially observable Markov decision process.

[0037] It can be understood that partially observable Markov decision processes are a framework for modeling decision problems, applicable to complex environments where states are not fully observable or deterministic. However, in real-world interactive question-answering tasks, agents can only make inferences based on past observations and historical decisions. Therefore, constructing an agent policy model using partially observable Markov decision processes can effectively model this uncertainty and partial observability, enabling agents to make effective decisions and plans even with incomplete information.

[0038] Step 202: obtain user input instructions, use the agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectories, and obtain a preference data set labeled with Q values.

[0039] It can be understood that in a given environment, the user inputs a command, and the agent's policy model generates an action (including an inference result and an executable prompt) based on the command. The agent then transitions state based on the action, integrates relevant information resources, and continues to interact with the user, generating an information search trajectory until the task is completed or the maximum time step is reached.

[0040] Simultaneously, when generating information search trajectories, a decision tree is constructed using the Monte Carlo Tree Search algorithm to inform the large-model agent's action selection during the decision-making process. The Monte Carlo Tree Search algorithm is a heuristic search algorithm primarily used for optimization problems in decision-making. Through simulation and optimization, it efficiently finds the optimal action during the question-answering process. By combining the agent's policy model with the Monte Carlo Tree Search algorithm and utilizing the node values ​​of the decision tree as supervised preference data, a high-quality preference dataset with Q-value annotations can be generated, providing a foundation for subsequent large-scale language model training.

[0041] Step 203: Set a large language model for learning Q values, and train the large language model using a step-level optimization objective function based on the preference data set labeled with Q values ​​to obtain a trained large language model.

[0042] Step 204: construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model.

[0043] Step 205 : Select actions based on the Q-value estimation and the information search trajectory to generate a question-answering process.

[0044] Understandably, traditional methods for enhancing the capabilities of large-scale intelligent agents, such as prompt-based methods, typically use prompt-based mechanisms to improve the agent's capabilities. This approach relies on manual intervention and annotation. However, as the complexity of the task increases, the difficulty of designing appropriate prompts also increases. Fine-tuning-based methods generally use rejection sampling fine-tuning, reinforcement learning (RL), or its variants to fine-tune large models using trajectories with result feedback. During the fine-tuning process, instability is easily encountered, generalization ability is poor, and the computational process requires a large amount of computing resources and is complex.

[0045] Based on this, the present invention establishes an additional large language model for learning Q-values ​​and trains it using a Q-value-annotated preference dataset, eliminating the need for manual annotation. A Q-value model is then constructed based on the trained large language model. This Q-value model has better generalization capabilities and can accurately assess the value of each action in the entire trajectory process, thereby improving the decision-making ability and task execution performance of the large model agent, enabling it to complete reasoning and decision-making tasks more accurately and efficiently in complex environments.

[0046] For a Q-value model built using a trained large language model, the large model agent can generate the action with the highest Q-value that interacts with the user. This is done by sampling multiple candidate actions during the decision tree expansion phase, then selecting an action based on the Q-value interaction with the user, and generating the result corresponding to the user input command, ultimately creating a question-answering process.

[0047] In the above-mentioned large-model intelligent agent interactive question-answering task decision-making method, an intelligent agent strategy model is first constructed based on a partially observable Markov decision process; then, user input instructions are obtained, and the intelligent agent strategy model is used to generate an information search trajectory; a decision tree of the information search trajectory is constructed using the Monte Carlo tree search algorithm to obtain a preference data set labeled with Q values; in addition, a large language model for learning Q values ​​is set up, and the large language model is trained using a step-level optimization objective function based on the preference data set labeled with Q values ​​to obtain a trained large language model; a Q-value model is constructed based on the trained large language model, and then a Q-value estimate is output based on the Q-value model; actions are selected based on the Q-value estimate and the information search trajectory to generate a question-answering process.

[0048] The present invention combines a large-model intelligent agent with a Monte Carlo tree search algorithm. Without the need to assume that the environment can be traced back and without manual annotation, it can obtain a high-quality Q-value labeled preference data set, thereby effectively supporting the exploration of high-quality question-answering interaction trajectories. At the same time, by adding a large language model and using the Q-value labeled preference data set and step-level optimization objective function to train the large language model, it can well refine the strategic improvement of the information search trajectory. Then, the Q-value model is constructed by the trained large language model, which can significantly improve the performance of the large-model intelligent agent, so that it can also have good performance in complex tasks. In addition, the present invention does not need to fine-tune the backbone of the intelligent agent's large model. Instead, it constructs a Q-value model by the trained large language model to evaluate each step of the trajectory and then selects the action accordingly, which significantly enhances the performance and generalization ability of the large-model intelligent agent in the autonomous reasoning and decision-making process. In addition, since there is no need to fine-tune the backbone of the intelligent agent's large model, the method proposed by the present invention can be efficiently combined with other large models, showing good compatibility and adaptability.

[0049] In one embodiment, an agent strategy model is constructed based on a partially observable Markov decision process, including:

[0050] According to the given environment, the agent interactive question-answering task is converted into a partially observable Markov decision process, which can be expressed as:

[0051] The agent strategy model is constructed based on the partially observable Markov decision process, which is expressed as:

[0052]

[0053] Where, Represents all possible user input instructions, that is, all types of queries that the agent needs to handle, such as user queries or questions; Represents all possible states encountered during the question-answering process, such as the agent's understanding of the question, known information, retrieved documents, and the current state of the derived answer; Represents the set of all possible actions that the agent can take in each state, such as asking questions, retrieving information, integrating information, reasoning, or generating part of the answer; Represents all the information resources available to the agent, including user feedback, retrieved document fragments, and related information extraction results; represents the probability of updating the state of the question understanding or answer after the agent performs an action (such as retrieval or reasoning); R represents the reward function, which is a binary reward based on the correctness of the answer (correct or wrong); π represents the agent's policy function; u represents the user input command in a given environment; τ represents the information search trajectory; a trepresents the action; t represents the current time step; T represents the total number of action steps; τ t-1 represents the trajectory of the previous time step.

[0054] In one embodiment, obtaining a user input instruction and using an agent strategy model to generate an information search trajectory includes:

[0055] Get the user input command u in a given environment and generate the information search trajectory at the current time step t. The information search trajectory expression is:

[0056] τ t =(a0,o0,...,a t ,o t )~π(τ t |u);

[0057] In the formula, o t Represents the information resources obtained by the agent at the current time step t.

[0058] It can be understood that in a partially observable Markov decision process In the function Expressed as Function R is expressed as

[0059] Specifically, for a user input instruction u in a given environment, the large model agent generates an action a1~π(·|u) according to its strategy π, and the agent state is transferred to The agent observes Then continue to interact with the user until the task is completed or the maximum time step is reached. At time step t, given the history and current observation, the agent generates the subsequent action a t+1 ~π(·|u,a0,o0,...,a t ,o t Based on this, we construct an agent strategy model for multi-step interactive question-answering tasks and generate information search trajectories at time step t. It is worth noting that action a t It contains a reasoning thought and an actionable hint.

[0060] In one embodiment, a Monte Carlo tree search algorithm is used to construct a decision tree of the information search trajectory to obtain a preference data set labeled with Q values, including:

[0061] The nodes of the decision tree represent states, the edges represent actions, and a set of statistics is stored at each node;

[0062] Start expanding from the root node. At each depth, calculate the upper confidence bound based on the statistical data stored in each node to select the appropriate leaf node for the next trajectory expansion.

[0063] When performing the next trajectory extension, several actions a are sampled from the information search trajectory. t Expand the leaf nodes until the maximum number of action steps is reached to generate a decision tree;

[0064] Obtain the preference dataset with Q value annotation according to the decision tree.

[0065] In one embodiment, when generating a decision tree, the method further includes: evaluating the expanded nodes through a rollout algorithm to generate an evaluation result; and backpropagating and updating the node data of the decision tree according to the evaluation result to obtain a decision tree.

[0066] Specifically, when generating information search trajectories, each planned path consists of several steps, so obtaining the value estimate of each intermediate step is of great significance. The present invention uses the Monte Carlo tree search algorithm to construct a decision tree, and defines that each node in the decision tree represents a state s, and each edge represents an action a. t Since in partially observable Markov decision processes, the complete state cannot be obtained at each step Only some of the observed Therefore, for the decision tree, the root node s0 contains (u), and the other nodes use the currently observed In addition, each node stores a set of statistics:

[0067]

[0068] Where V(s) represents the value function (expected return) of the subtree from node s, Indicates the number of times node s is visited.

[0069] After determining the framework of the decision tree, we begin to expand the tree. First, we select the trajectory from the root node to the leaf node. At each depth of the decision tree, we calculate the upper confidence bound value based on the statistical data stored in each node to select the appropriate leaf node for the next trajectory expansion. The expression of the upper confidence bound is:

[0070]

[0071] Where w represents the exploration weight and p(s) is the parent node of node s.

[0072] The upper confidence bound can be used to balance exploration and utilization.

[0073] Then, by sampling n actions a from the information search trajectory tExpand the leaf nodes. It can be understood that, unlike the action space of traditional agents, which is usually limited, the action space of large model agents is infinite. Therefore, large models can generate an infinite number of unique actions (token sequences), even if they are invalid. Based on this, the present invention uses high temperature to sample multiple actions to ensure diversity, and the environment processes each action a t , and provide corresponding feedback as observation, generate n new leaf nodes as extensions, and incorporate them into the tree.

[0074] However, the decision tree constructed in this way is shallow, and the expansion process quickly reaches the terminal node. Therefore, the expanded nodes are evaluated using a rollout algorithm. Specifically, starting from the expanded leaf node, the large model agent interacts with the user until the interaction ends or the maximum rollout depth is reached. If the explored node reaches the goal, the result is a reward; otherwise, the node explored at the maximum depth is set to a fixed negative reward.

[0075] After the evaluation is completed, the node data of the tree is updated through back propagation. The update is based on the evaluation results of the evaluation phase, that is, the result is a reward or a fixed negative reward. t For each node in, N(s) increases by 1, and the value is transferred from the terminal node s T Update to the root node s0, and the expression for backpropagation update is:

[0076]

[0077] The updated value is used in the UCT formula to guide the selection of the next node.

[0078] Finally, after multiple iterations of selection, expansion, evaluation, and backpropagation, a decision tree is obtained. Specifically, according to the node reward r(s), the best leaf node for the agent to explore is first found, and then the corresponding trajectory is obtained in a top-down manner. At each depth, a part of the best trajectory is taken as the unfinished trajectory segment τ T . Then the preferred action for the next step is Taken from the best trajectory, with less preferred actions Select the candidate action with the minimum value, and the preference data marked by Q value is as follows Figure 2 As shown in Figure 2. The white nodes represent the best trajectories explored by the agent and are considered as the selected nodes at each depth of the tree. The gray nodes are candidates for constructing less preferred samples, while the black nodes are ignored.

[0079] It can be understood that the preference data labeled with Q values ​​refers to the preference of the data in the dataset reflected by the Q value. By continuously training the large language model through iterative cycles and further updating the Q value model, it is equivalent to continuously training the Q value model through the preference data labeled with Q values. Therefore, the actions in the entire information search trajectory are affected by the Q value. The actions with higher Q value estimates will be more "preferred" and become the optimal actions, which in turn helps the large model intelligent agent generate a question-answering process that is more in line with the specific environment.

[0080] By using step-level preference data to train a large language model, the constructed Q-value model can have a significant improvement in sample efficiency. Figure 3 , shows the impact of the number of step-level preference data on model performance, which evaluates the effect of different numbers of training samples on the performance of the Q-value model on the HotPotQA task. In evaluating the HotPotQA task, several checkpoints of a cycle are trained by training a large language model and updating the Q-value model. These checkpoints represent different numbers of training samples. It can be seen that less than 400 step-level preference data points can significantly improve performance, and only about 250 task instructions are required to implement it. This proves that by training a large language model with step-level preference data, the Q-value model constructed based on the trained large language model can significantly improve sample efficiency. Therefore, the efficiency and effect of the constructed Q-value model can be understood through different numbers of step-level preference data, so as to select the appropriate number.

[0081] See Figure 4 , shows the results of constructing a Q-value labeled preference dataset using different numbers of Monte Carlo tree search algorithm iterations. It analyzes the effect of increasing the number of Monte Carlo tree search algorithm iterations on the number of successful trajectories collected and the number of step-level preference data points. Figure 4 It is possible to understand the impact of setting the number of Monte Carlo Tree Search iterations during data collection on the size and quality of the final dataset. Specifically, the number of successful trajectories used to construct step-level preference data increases with the number of Monte Carlo Tree Search iterations. Almost all Monte Carlo Tree Search processes terminate early, before reaching maximum reward or depth, typically before the 50th iteration, making additional iterations redundant. Furthermore, as the number of Monte Carlo Tree Search iterations increases, the number of step-level preference data points increases faster than the number of successful trajectories. This is because trajectories explored through more Monte Carlo Tree Search iterations typically involve more decision steps, thus providing more step-level preference data.

[0082] In one embodiment, a large language model is trained using a step-level optimization objective function based on a preference dataset labeled with Q values ​​to obtain a trained large language model, including:

[0083] Preference dataset labeled by Q value Get preference data

[0084] Then the expression of the step-level optimization objective function is:

[0085]

[0086] Where β represents the weight of the constraint; π Q Represents a large language model; represents the reference model; Actions that indicate best preferences; represents the less preferred action; u represents the user input instruction in a given environment; τ t represents the information search trajectory at the current time step t; represents the averaging operation; σ represents the activation function.

[0087] In one embodiment, the expression of the Q value model is:

[0088]

[0089] It can be understood that, unlike the traditional direct policy optimization (DPO) which targets the trajectory level, the present invention utilizes a more fine-grained Q-value labeled preference dataset. Constructing step-level optimization objective function, we implement direct policy optimization (DPO) at the step level, which provides more detailed and accurate results by observing specific steps more deeply, making the large language model π Q This has better generalization performance, which in turn enables the Q-value model to have better generalization performance. In addition, by additionally training a large language model to build the Q-value model, rather than fine-tuning the large model backbone of the intelligent agent, the method proposed in this invention can be integrated with both open source and API-based large-model intelligent agents.

[0090] In one of the embodiments, the superiority of the proposed method is verified. The proposed method is marked with (Ours), and the dataset uses HotPotQA in an interactive question-answering scenario. HotPotQA is a question answering task that requires searching in Wikipedia articles. The large model agent is equipped with API calls for searching and retrieving information. After receiving the answer, the environment provides a binary reward of 0 or 1 based on its correctness. The maximum number of steps is set to 7. When the agent chooses the "buy" action, the environment provides a final reward ranging from 0 to 1 based on the matching heuristic of product attributes and prices. The maximum number of steps is set to 10. All methods in the experiment use the ReAct-style (Yao et al., 2023b) interaction format, generate CoT (Wei et al., 2022) thinking before the action, the maximum context length is set to 4096, and the number of candidate actions is set to n=5.

[0091] In addition, the effectiveness of the proposed method is also verified in an automated shopping scenario. The WebShop task requires the agent to solve a shopping task by browsing a website with detailed product descriptions and specifications. The available action APIs include SEARCH[QUERY] for using the search bar and CHOOSE[BUTTON] for clicking buttons on the web page. Clickable buttons include product title, options, purchase, return to search, and previous / next page. When the user selects the "buy" action, the environment provides a final reward ranging from 0 to 1 based on the matching heuristic of product attributes and prices.

[0092] Training Settings: Before collecting step-level preference data, the maximum number of iterations of the Monte Carlo tree search was set to 30. When building the Q-value model, Phi-1.5 was primarily used for large language models. In the HotPotQA task, 1000 questions were randomly selected for training, 100 questions were used for validation, and 100 questions were used for testing. In the WebShop task, the data split was based on Song et al. (2024), with 1824 instructions for training and 200 instructions for testing. The batch size was set to 16, the learning rate was set to 1e-5, the warm-up ratio was 0.1, a cosine scheduler was used, and β in the DPO loss was set to 0.1. All experiments (except for the fine-tuning-based comparison method) were conducted on an NVIDIA A4048G GPU. The experimental results are shown in Table 1.

[0093]

[0094] It can be seen that the Q-value model proposed in the present invention can significantly enhance decision-making capabilities. The Q-value model proposed in the present invention performs twice as well in the WebShop task as the large model agent established by Phi-3-mini-4k-instruct, and improves performance by 75% in the HotPotQA task. Since BoN is a strong baseline, it has the same computational overhead as the method proposed in the present invention, but has additional result rewards from the environment. However, the method proposed in the present invention outperforms BoN in both tasks. Compared with the most advanced commercial large-model agent based on API, the method proposed in the present invention surpasses the lightweight API-based GPT-4o-mini in both tasks, and even surpasses the most effective GPT-4-turbo in the WebShop task.

[0095] Building a Q-value model by training a large language model is more efficient and effective than fine-tuning a large language model (LLM) backbone. Using demonstrated trajectories to supervise the RFT fine-tuning of the LLM improves performance on agent tasks. Furthermore, AgentEval, which uses more reward information, further improves performance on the WebShop task, but not on the HotPotQA task. This is because the HotPotQA environment only provides binary rewards. ETO, which adds more negative examples for learning, achieves the best results among fine-tuning-based methods. This highlights the need to incorporate imperfect trajectories into training. Fine-tuning the LLM backbone requires high-performance computing resources, especially when the LLM size and context length are large. Therefore, the comparison of our method with fine-tuning-based methods is primarily based on the Phi-3-mini-4k-instruct with 3.8B parameters. In contrast, our Q-value model is built using the more lightweight Phi-1.5 with 1.3B parameters. Despite this, our method is more effective than all of the aforementioned fine-tuning-based methods.

[0096] The Q-value model constructed by the present invention can be generalized to different LLM backbones. The Q-value function simulates the future total reward in the environment. Therefore, it can be generalized to different large-model agents within the task. To verify this, the preference data sampled by Phi-3-mini-4k-instruct was first used to train a large language model and build a Q-value model. Then, the constructed Q-value model was directly applied to the more powerful open source large-model Llama-3.1-8B-instruct and API-based LLMs, including GPT-4o-mini and GPT-4-turbo. Even if the performance improvement is not as high as when applied to the large-model agent with sampled training data, the decision-making ability can be continuously improved. This is because the actions sampled by the more powerful large-model agent can be regarded as out-of-distribution (OOD) relative to the step-level preference data collected by Phi-3-mini-4k-instruct for training the large language model and then building the Q-value model. This is a promising way to use the trial-and-error experience of weak and cheap large-model agents to assist more powerful or expensive API-based large-model agents.

[0097] Furthermore, the accuracy of the Q-value model constructed by the present invention in evaluating the preference relationship of the collected step-level data was also studied. Figure 5 As shown in (a), it can be observed that the preference relationship in the training set can be easily learned in both tasks. The accuracy of the IND and OOD test sets decreases in turn, which is about 80% in the WebShop task and about 65% in the HotPotQA task. As mentioned above, the result reward provided by the HotPotQA environment is binary. In addition to early stopping when reward 1 is obtained, as shown in Figure 5 As shown in (b), the Q-values ​​of the nodes are as distinct as those in the WebShop task. This suggests designing more complex rewards to further improve performance. Nevertheless, this preference accuracy is sufficient to improve performance on downstream tasks.

[0098] To further evaluate the effectiveness of the Q-value model constructed by the present invention, we randomly selected 200 actions from the successful and failed trajectories, respectively, and visualized their Q-values ​​in the figure. All trajectories were sampled from the test instructions of the WebShop task by Phi-3-mini-4k-instruct. The Q-value distribution of actions in the failed trajectories is skewed to the left. In contrast, the distribution of incorrect solutions shows a smaller slope, with most probability densities skewed to the right. This pattern may be because failures usually result from the selection of some destructive actions, which shows that the Q-value model constructed by the present invention has the ability to assign credit.

[0099] It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0100] In one embodiment, Figure 6 As shown, a large-model intelligent agent interactive question-answering task decision-making device is provided, comprising: an intelligent agent strategy model construction module 401, a preference data set generation module 402, a large language model training module 403, a Q-value estimation calculation module 404, and a question-answering process generation module 405, wherein:

[0101] The intelligent agent strategy model construction module 401 is used to construct an intelligent agent strategy model based on a partially observable Markov decision process.

[0102] The preference data set generation module 402 is used to obtain user input instructions, generate information search trajectories using the intelligent agent strategy model, and construct a decision tree of the information search trajectory using the Monte Carlo tree search algorithm to obtain a preference data set labeled with Q values.

[0103] The large language model training module 403 is used to set a large language model for learning Q values, and train the large language model using a step-level optimization objective function based on the preference data set labeled with Q values ​​to obtain a trained large language model.

[0104] The Q-value estimation calculation module 404 is used to construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model.

[0105] The question-answering process generating module 405 is used to select actions based on the Q-value estimation and the information search trajectory to generate a question-answering process.

[0106] Regarding the specific limitations of the large-model intelligent agent interactive question-answering task decision-making device, please refer to the limitations of the large-model intelligent agent interactive question-answering task decision-making method above, which will not be repeated here. The various modules in the above-mentioned large-model intelligent agent interactive question-answering task decision-making device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0107] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store decision data for large-model intelligent agent interactive question-answering tasks. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a large-model intelligent agent interactive question-answering task decision method is implemented.

[0108] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0109] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0110] Step 201: construct an agent strategy model based on a partially observable Markov decision process.

[0111] Step 202: obtain user input instructions, use the agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectories, and obtain a preference data set labeled with Q values.

[0112] Step 203: Set a large language model for learning Q values, and train the large language model using a step-level optimization objective function based on the preference data set labeled with Q values ​​to obtain a trained large language model.

[0113] Step 204: construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model.

[0114] Step 205 : Select actions based on the Q-value estimation and the information search trajectory to generate a question-answering process.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0116] Step 201: construct an agent strategy model based on a partially observable Markov decision process.

[0117] Step 202: obtain user input instructions, use the agent strategy model to generate information search trajectories; use the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectories, and obtain a preference data set labeled with Q values.

[0118] Step 203: Set a large language model for learning Q values, and train the large language model using a step-level optimization objective function based on the preference data set labeled with Q values ​​to obtain a trained large language model.

[0119] In step 204, a Q-value model is constructed based on the trained large language model, and then a Q-value estimate is output based on the Q-value model. In step 205, an action is selected based on the Q-value estimate and the information search trajectory to generate a question-answering process.

[0120] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0121] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-described embodiments merely illustrate several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that a person skilled in the art would be able to make numerous modifications and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A large-scale intelligent agent interactive question-answering task decision-making method, characterized by: The method comprises: Build agent strategy models based on partially observable Markov decision processes; Obtaining user input instructions, using the agent strategy model to generate an information search trajectory; using the Monte Carlo tree search algorithm to construct a decision tree of the information search trajectory, and obtaining a preference data set annotated with Q values; Setting a large language model for learning Q values, and training the large language model using a step-level optimization objective function according to the preference data set labeled with the Q values ​​to obtain a trained large language model; Build a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model; Selecting an action based on the Q-value estimate and the information search trajectory to generate a question-answering process; A decision tree of the information search trajectory is constructed using the Monte Carlo tree search algorithm to obtain a preference data set labeled with Q values, including: The nodes of the decision tree represent states, the edges represent actions, and a set of statistics is stored at each node; Start expanding from the root node. At each depth, calculate the upper confidence bound based on the statistical data stored in each node to select the appropriate leaf node for the next trajectory expansion. When performing the next trajectory expansion, several actions are sampled from the information search trajectory. Expand the leaf nodes until the maximum number of action steps is reached to generate a decision tree; Obtaining a preference data set labeled with Q values ​​according to the decision tree; According to the preference data set annotated with the Q value, the large language model is trained using a step-level optimization objective function to obtain a trained large language model, including: Preference dataset labeled according to the Q value , get preference data pair ; Then the expression of the step-level optimization objective function is: ； Where, represents the weight of the constraint item; Represents a large language model; represents the reference model; Actions that indicate best preferences; Indicates less preferred action; Represents user input instructions in a given environment; Indicates the current time step The information search trajectory below; Indicates averaging operation; Represents the activation function.

2. The large-scale intelligent agent interactive question-answering task decision-making method according to claim 1 is characterized in that: Build an agent strategy model based on a partially observable Markov decision process, including: According to the given environment, the agent interactive question-answering task is converted into a partially observable Markov decision process, which can be expressed as: ; The agent strategy model is constructed based on the partially observable Markov decision process, which is expressed as: ； Where, Represents all possible user input commands; Represents all possible states that may be encountered during the question-answering process; represents the set of all possible actions that the agent can take in each state; Represents all the information resources available to the agent; Represents the probability of updating the question understanding or answer status after the agent performs an action; represents the reward function; represents the agent’s policy function; Represents user input instructions in a given environment; represents the information search trajectory; Indicates action; Indicates the current time step; Indicates the total number of action steps; represents the trajectory of the previous time step.

3. The large-scale intelligent agent interactive question-answering task decision-making method according to claim 2 is characterized in that: Obtaining user input instructions and using the agent strategy model to generate an information search trajectory includes: Get user input instructions in a given environment , generates the current time step The information search trajectory under the following conditions is expressed as: ； Where, Indicates the current time step The information resources obtained by the agent.

4. The large-scale intelligent agent interactive question-answering task decision-making method according to any one of claims 1 to 3, characterized in that: When generating a decision tree, it also includes: Evaluate the expanded nodes through the rollout algorithm and generate evaluation results; The node data of the decision tree is updated by back propagation according to the evaluation result to obtain a decision tree.

5. The large-scale intelligent agent interactive question-answering task decision-making method according to any one of claims 1 to 3, characterized in that: The expression of the Q value model is: 。 6. A large-scale intelligent agent interactive question-answering task decision-making device, characterized in that: The large-model intelligent agent interactive question-answering task decision method according to any one of claims 1 to 5 is adopted, wherein the device comprises: An agent policy model building module, which is used to build an agent policy model based on a partially observable Markov decision process; A preference data set generation module is used to obtain user input instructions and use the agent strategy model to generate an information search trajectory; a Monte Carlo tree search algorithm is used to construct a decision tree of the information search trajectory to obtain a preference data set annotated with a Q value; A large language model training module is used to set a large language model for learning Q values, and train the large language model using a step-level optimization objective function based on the preference data set labeled with the Q values ​​to obtain a trained large language model; A Q-value estimation calculation module is used to construct a Q-value model based on the trained large language model, and then output a Q-value estimate based on the Q-value model; The question-answering process generation module is used to select actions based on the Q-value estimation and the information search trajectory to generate a question-answering process.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the large-model intelligent agent interactive question-answering task decision-making method described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large-model intelligent agent interactive question-answering task decision-making method described in any one of claims 1 to 5 are implemented.

Images