Training methods and related devices for intelligent agents

Abstract

This application provides a method and related apparatus for training an intelligent agent. The method includes: acquiring dialogue trajectory data during a simulated dialogue between a first intelligent agent and a second intelligent agent; wherein the first intelligent agent and the second intelligent agent respectively imitate different roles; determining target dialogue trajectory data based on the quality score of the dialogue trajectory data; wherein the quality score is used to represent the degree of fit between user questions and answers in the dialogue trajectory data; the degree of fit between user questions and answers in the target dialogue trajectory data is higher than the degree of fit between user questions and answers in non-target dialogue trajectory data; and training a target intelligent agent based on the target dialogue trajectory data. This application can improve the training efficiency of the intelligent agent to a certain extent, as well as enhance the agent's tool invocation and dialogue reasoning capabilities.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for training an intelligent agent. Background Technology

[0002] In the field of large-model-based agent training, training data typically comes from manually written examples and question-and-answer samples generated from real-world transaction dialogue records.

[0003] However, intelligent agents require a large number of question-and-answer samples to meet training needs. Generating question-and-answer samples based on manually written examples or real-world transaction dialogue records is inefficient. Summary of the Invention

[0004] In view of this, one or more embodiments of this application provide a training method and related apparatus for an intelligent agent, which can improve the training efficiency of the intelligent agent.

[0005] In a first aspect, one or more embodiments of this application propose a method for training an intelligent agent, comprising: acquiring dialogue trajectory data during a simulated dialogue between a first intelligent agent and a second intelligent agent; wherein the first intelligent agent and the second intelligent agent respectively imitate different identity roles; the dialogue trajectory data includes at least a user question generated by the first intelligent agent, a reasoning chain of the second intelligent agent corresponding to the user question, and a question answer generated by the second intelligent agent based on the reasoning chain; the reasoning chain includes multiple reasoning tasks, and the multiple reasoning tasks are used to characterize the reasoning process of the second intelligent agent in responding to the user question and generating a question answer; determining target dialogue trajectory data based on the quality score of the dialogue trajectory data; wherein the quality score is used to represent the degree of fit between the user question and the question answer in the dialogue trajectory data; the degree of fit between the user question and the question answer in the target dialogue trajectory data is higher than the degree of fit between the user question and the question answer in non-target dialogue trajectory data; and training a target intelligent agent based on the target dialogue trajectory data.

[0006] Secondly, one or more embodiments of this application propose a training apparatus for an intelligent agent, comprising: an acquisition module, configured to acquire dialogue trajectory data during a simulated dialogue between a first intelligent agent and a second intelligent agent; wherein the first intelligent agent and the second intelligent agent respectively imitate different identity roles; the dialogue trajectory data includes at least a user question generated by the first intelligent agent, a reasoning chain of the second intelligent agent corresponding to the user question, and a question answer generated by the second intelligent agent based on the reasoning chain; the reasoning chain includes multiple reasoning tasks, and the multiple reasoning tasks are used to characterize the reasoning process of the second intelligent agent in responding to the user question and generating a question answer; a determination module, configured to determine target dialogue trajectory data based on the quality score of the dialogue trajectory data; wherein the quality score is used to represent the degree of fit between the user question and the question answer in the dialogue trajectory data; the degree of fit between the user question and the question answer in the target dialogue trajectory data is higher than the degree of fit between the user question and the question answer in non-target dialogue trajectory data; and a training module, configured to train a target intelligent agent based on the target dialogue trajectory data.

[0007] Thirdly, one or more embodiments of this application provide a computer device including a memory and a processor, wherein the memory stores at least one computer program, which is loaded and executed by the processor to implement the method as described above.

[0008] Fourthly, one or more embodiments of this application provide a computer program product including computer instructions that, when executed by a processor, implement the method as described above.

[0009] Fifthly, one or more embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method as described above.

[0010] As can be seen from the above embodiments, multiple embodiments of this application obtain dialogue trajectory data generated by the first intelligent agent and the second intelligent agent during the simulated dialogue process, and use the degree of fit between user questions, inference chains and question answers in the dialogue trajectory data to determine high-quality target dialogue trajectory data, and then train the target intelligent agent based on the target dialogue trajectory data, thereby realizing the automatic screening and utilization of high-quality training samples and achieving the effect of improving the training efficiency of the intelligent agent. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of a training method for an intelligent agent provided in one embodiment of this application.

[0012] Figure 2This is a schematic diagram of a training method for an intelligent agent provided in one embodiment of this application.

[0013] Figure 3 This is a schematic diagram of a single intelligent agent interacting with a user, provided in one embodiment of this application.

[0014] Figure 4 This is a flowchart illustrating a training method for an intelligent agent provided in one embodiment of this application.

[0015] Figure 5 This is a schematic diagram of a module of a training device for an intelligent agent provided in one embodiment of this application.

[0016] Figure 6 This is a schematic diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0017] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments.

[0018] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0019] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0020] In related technologies, in large-model-based agent training scenarios, it is common practice to rely on manually writing examples or compiling question-and-answer samples from real-world dialogue records to provide basic data for agent training. However, the process of constructing such training data requires a significant amount of manual input, which is not only inefficient but also makes it difficult to generate a sufficient number of training samples covering diverse scenarios in a short period of time. This results in a long agent training cycle, making it difficult to meet the needs of continuous agent upgrades and rapid iterations.

[0021] With the continuous improvement of generative large models, automatically generating dialogue samples using generative large models has become an effective way to replace manually constructed data. However, dialogue content generated by a single generative large model may still have problems such as unstable quality, opaque reasoning process, and fluctuating fit between answers and questions, making it difficult to use directly as high-quality training data.

[0022] In summary, there is still a lack of a method in the relevant technologies that can efficiently generate and automatically filter high-quality dialogue samples for training agents, and further improvements are urgently needed.

[0023] In several embodiments provided in this application, the training method for intelligent agents can be applied to electronic devices with certain computing and network access capabilities. This electronic device can be a desktop computer, laptop computer, tablet computer, smartphone, or a server. Specifically, the electronic device includes a processor, memory, and a network access module for network communication. The server can be an electronic device with strong data processing capabilities; of course, a server can also refer to a server cluster formed by multiple electronic devices, or a quantum server built using a quantum computer.

[0024] Please see Figure 1 This application provides an example application scenario for a training method for intelligent agents. In this example, a medical consultation scenario is used. The training device can first extract identity information and dialogue behavior data from real-world doctor-patient consultation data. For example, real dialogue data may contain language behavior characteristics of different roles such as patient consultants, doctors, or health advisors, such as "patients typically describe symptoms and express concerns" and "doctors habitually confirm key information before giving a grading assessment." Based on this identity information and dialogue behavior data, the training device can construct a first intelligent agent (simulating patient questioning behavior) and a second intelligent agent (simulating a medical consultant).

[0025] After constructing the two agents, the training device can instruct the first agent to engage in simulated dialogue with the second agent. For example, the first agent generates a user question: "My child has been coughing for two weeks, does he need medication?" The second agent generates a reasoning chain based on its internal reasoning ability, such as: Reasoning Task 1: Recognizing that "coughing for two weeks" is a long-term symptom; Reasoning Task 2: Determining the potential risk of infection; Reasoning Task 3: Inferring the need for tiered medical advice; Reasoning Task 4: Generating an answer to the question: "If the cough persists for more than two weeks, it is recommended to have basic examinations first before deciding whether to use medication."

[0026] The training device can generate dialogue trajectory data based on the simulated dialogue process and assign quality scores to the degree of fit between user questions and answers. The training device will automatically select high-quality target dialogue trajectory data and filter out dialogue samples with incoherent reasoning or insufficient answers.

[0027] When training a target agent using target dialogue trajectory data, the training device can construct a reasoning tree based on the reasoning chain. For example, using "identifying symptom types" as the root node in the reasoning chain, the training device adds different types of pre-defined reasoning tasks to the reasoning tree, including needs understanding tasks, action planning and tool invocation tasks, proactive dialogue tasks, knowledge reasoning and computation tasks, solution recommendation and product marketing tasks, anthropomorphic response generation tasks, and reflection and self-evolution tasks. The training device can score each task node in the reasoning tree and traverse the tree, selecting the reasoning path with the higher node score as the first target reasoning chain. Simultaneously, the training device can also select reasoning paths with lower node scores and weaker logic as the second target reasoning chain, used to clearly distinguish between high-quality and low-quality reasoning during training.

[0028] The training device can further train the target agent using the first and second target inference chains, causing the target agent's inference chains to gradually gravitate towards the first target inference chain and avoid the second target inference chain. Subsequently, the training device can input these two inference chains into a designated agent with a larger parameter scale for learning, enabling the designated agent to gain richer inference experience. Then, through distillation learning, the inference preferences from the large model are transferred to the target agent, thereby improving the target agent's inference ability on a small model scale.

[0029] Please see Figure 2 In another scenario example, the training device can further perform various types of specified reasoning tasks based on the target dialogue trajectory data to generate target training samples. For example, the training device can use a task execution module to perform tasks such as need understanding (e.g., identifying "concerns about safety" from "Can my child use this?"), action planning and tool invocation (e.g., planning "first check the applicable population of the drug" and then "obtain relevant product information"), proactive dialogue (e.g., asking "How old is your child?" to guide the dialogue), knowledge reasoning and computation (e.g., inferring the duration of action based on knowledge about the drug), solution recommendation (e.g., providing "alternative product combinations" based on user needs), anthropomorphic response generation (e.g., generating "more emotionally valuable" reassuring expressions), and reflection and self-evolution (e.g., performing "risk checks and logical corrections" on the entire response). Multiple specified reasoning tasks can correspond to the same task execution module, or each specified reasoning task can correspond to a separate task execution module.

[0030] The training device can collect reasoning information and task results for each specified reasoning task, and generate a task evaluation score based on the degree of matching between the task results and real dialogue data. Combining the reasoning information, task results, and task evaluation score, the reasoning module performs comprehensive reasoning processing to construct target training samples. Target training samples may include agent function definitions (e.g., "possessing high-quality medical consultation reasoning ability"), training execution data (e.g., user symptom descriptions), and sample reasoning chains (e.g., task combinations under the target reasoning path).

[0031] During the training of the target agent using target training samples, the training device can obtain feedback signals from different dimensions through a reward model. For example, it evaluates the degree of matching between the current question answer and the standard answer based on the first reward function, evaluates the consistency between the inference chain generated by the target agent and the sample inference chain based on the second reward function, and evaluates the fit between the current question answer and the inference chain based on the third reward function. Based on these three feedback signals, the target agent is comprehensively adjusted to gradually improve the correctness of the answer content, the logic of the inference chain, and the consistency between the answer and the inference process.

[0032] Through the above training process, the training device can automatically generate, filter, fuse, and enhance high-quality training samples, enabling the target intelligent agent to obtain higher reasoning ability and transaction performance in continuous iteration.

[0033] Please see Figure 3 In another scenario example of this application, in a customer consultation scenario, the e-commerce platform can deploy a target intelligent agent to automatically respond to user inquiries about medical health, products, or general medical consultations. The e-commerce platform can use a single target intelligent agent to identify, reason about, invoke tools, and answer user questions.

[0034] In this scenario example, the e-commerce platform can be equipped with an intent-based audience extraction module to identify whether users express a desire to switch to a human agent in their inquiry text. For example, when a user enters statements such as "I want to directly contact a human customer service representative" or "I don't want to talk to a robot," the intent-based module can determine that the user wants to switch to a human agent and thus directly transfer the user's conversation to a human agent. When the user's statement does not contain such semantics, the user's question will be provided to the target AI agent for processing. For example, the user's question might be: "I've been experiencing a dull ache in my upper right abdomen after eating lately; could it be a gallbladder problem?" After a user's question is sent to the target agent, the agent can retrieve and organize relevant memory information to serve as the basis for subsequent reasoning and decision-making. Specifically, the target agent can retrieve user profile information such as the user's historical consultation records, past purchase or medication preferences, and long-term health concerns from the long-term memory module; retrieve the context of the current multi-turn dialogue, user-added symptom descriptions, time / location / triggers, and other immediate information from the short-term memory module; and retrieve intermediate reasoning conclusions formed by the target agent in the current dialogue, planned steps to be executed, and feedback from previously executed reasoning tasks from the planning memory module. The target agent can then fuse and align the memory information from these different sources to form a contextual semantic basis corresponding to the current user's question, thereby planning multiple specified reasoning tasks. These tasks are then executed by the task execution module, which provides feedback on the execution results of the specified reasoning tasks to the target agent.

[0035] In this scenario example, the target agent can initiate a demand understanding task, analyzing the user's input of "dull pain in the upper right abdomen" to identify the core demand as "determining the possible causes of the symptoms." Subsequently, the target agent can execute action planning and tool invocation tasks. Based on the results of the symptom analysis, it determines that the next step requires querying a medical knowledge base or performing symptom matching. Therefore, the target agent can use the action executor to invoke disease diagnosis tools or knowledge search tools to query common causes related to "upper right abdominal pain," such as cholecystitis, gallstones, and gastrointestinal dysfunction.

[0036] Simultaneously, the target agent can also perform proactive dialogue tasks to obtain supplementary information. For example, the target agent can generate a thought process based on planned memory: "It is necessary to further confirm whether the user is experiencing fever, nausea, or worsening symptoms after meals." Based on this thought process, the target agent can ask the user clarifying questions, such as: "Besides pain, do you also experience nausea, fever, or worsening symptoms after meals?" After obtaining supplementary information from the user, the target agent can continue to perform knowledge reasoning and computation tasks. For example, it can calculate the matching degree of typical symptoms of diseases such as cholecystitis and gallstones, and combine the user's symptoms with the conclusions of the medical knowledge base to derive a reasoning chain. In this example, the reasoning chain can be: "Right upper abdominal pain → mostly caused by gallbladder disease → if it worsens after meals, gallstones need to be ruled out."

[0037] Once the inference chain is generated, the target agent can further perform tasks such as recommending solutions and marketing products. For example, if the inference chain indicates that a user may have a risk of cholecystitis and the symptoms are mild, the target agent can recommend "online doctor consultation," "booking a physical examination," or health products related to gallbladder health based on the platform's strategy; if the inference chain indicates the presence of high-risk factors, the target agent can prompt the user to "seek medical attention as soon as possible."

[0038] Subsequently, the target agent can also perform the task of generating human-like responses, integrating the professional conclusions, risk warnings, and behavioral suggestions in the reasoning process into natural language answers, such as: "Your symptoms are indeed highly correlated with the gallbladder, especially the pain that worsens after meals. It is recommended that you go to the hospital as soon as possible for an abdominal ultrasound examination, and at the same time avoid high-fat diets." In this scenario example, the target agent can also perform reflection and self-evolution tasks internally. By comparing the current inference chain, the effect of tool calls, user feedback, and the consistency between the thinking process and the answer, it can automatically generate improvement signals for self-optimization in subsequent rounds of dialogue or training phases.

[0039] Additionally, e-commerce platforms can set up an exit recognition module to determine whether a user wishes to end the conversation. For example, when a user says "That's all, thank you" or "Ask again tomorrow," the exit recognition module can end the current conversation.

[0040] Throughout the scenario, the target intelligent agent in the e-commerce platform achieves automated processing of user health consultations or product needs through continuous reasoning, planning, tool invocation, and behavior generation.

[0041] Please see Figure 4 One embodiment of this application provides a method for training an intelligent agent. The method can be applied to a training device, which can be applied to the aforementioned electronic device possessing certain computing power and network access capabilities. Of course, in some embodiments, the training device can also be software running on the electronic device. The method for training an intelligent agent may include the following steps.

[0042] Step S110: Obtain dialogue trajectory data during the simulated dialogue between the first agent and the second agent; wherein the first agent and the second agent respectively imitate different identities; the dialogue trajectory data includes at least a user question generated by the first agent, a reasoning chain of the second agent corresponding to the user question, and a question answer generated by the second agent based on the reasoning chain; the reasoning chain includes multiple reasoning tasks, and the multiple reasoning tasks are used to characterize the reasoning process of the second agent in responding to the user question and generating a question answer.

[0043] Step S120: Determine the target dialogue trajectory data based on the quality score of the dialogue trajectory data; wherein, the quality score is used to represent the degree of fit between the user's question and the question answer in the dialogue trajectory data; the degree of fit between the user's question and the question answer in the target dialogue trajectory data is higher than the degree of fit between the user's question and the question answer in the non-target dialogue trajectory data.

[0044] Step S130: Train the target agent based on the target dialogue trajectory data.

[0045] In this embodiment, the training device can be used to acquire dialogue trajectory data of the first and second intelligent agents during the execution of a simulated dialogue, so as to further train the target intelligent agent based on the dialogue trajectory data. The first and second intelligent agents are used to imitate different identity roles to construct a simulated dialogue scenario with dialogue interaction characteristics. Specifically, the first intelligent agent can be used to generate user questions according to preset identity roles, and the second intelligent agent can be used to reason about the user questions based on the corresponding identity roles and generate question answers. While acquiring the user questions generated by the first intelligent agent and the question answers generated by the second intelligent agent, the training device can also acquire the reasoning chain used by the second intelligent agent when generating the question answers. The reasoning chain can be used to describe the reasoning structure of the second intelligent agent in the process of generating question answers and includes multiple reasoning tasks; the sequential relationship of the multiple reasoning tasks is used to characterize the reasoning path of the second intelligent agent from receiving user questions to outputting question answers.

[0046] In this embodiment, the training device can be used to evaluate the quality of the acquired dialogue trajectory data to determine the target dialogue trajectory data. Specifically, the training device can calculate a quality score for each dialogue trajectory data, which characterizes the degree of fit between the user's question and the question's answer in the dialogue trajectory data. The degree of fit can be used to characterize whether the question's answer accurately responds to the user's question and whether the corresponding reasoning chain can reasonably support the answer. In some embodiments, when the quality score of the dialogue trajectory data is higher than a preset threshold, the training device can determine the dialogue trajectory data as the target dialogue trajectory data; while dialogue trajectory data below the threshold can be regarded as non-target dialogue trajectory data and will not participate in subsequent training. Through the above screening process, the training device can automatically filter out the target dialogue trajectory data with a higher degree of fit between the user's question and the question's answer from multiple dialogue trajectory data, providing high-quality samples for agent training.

[0047] In this embodiment, the training device can train the target agent based on target dialogue trajectory data. Specifically, the training device can provide the target agent with user questions, reasoning chains, and question answers from the target dialogue trajectory data as training samples, enabling the target agent to learn the reasoning process and answering patterns of the second agent. During training, the target agent can gradually optimize its parameters based on the target dialogue trajectory data to improve its ability to handle user questions, construct reasoning chains, and generate question answers. In some embodiments, the training device can employ supervised learning or reinforcement learning to optimize the target agent, enabling it to achieve higher reasoning quality and answer accuracy in subsequent task execution.

[0048] Multiple embodiments of this application acquire dialogue trajectory data generated by a first agent and a second agent during a simulated dialogue, and determine high-quality target dialogue trajectory data by utilizing the degree of fit between user questions, inference chains and question answers in the dialogue trajectory data. Then, the target agent is trained based on the target dialogue trajectory data, thereby achieving automatic screening and utilization of high-quality training samples and improving the training efficiency of the agent.

[0049] In one specific implementation, to perform the aforementioned training method, the training device can model atomic capabilities and Markov decision processes (MDPs). Specifically, the state space is defined as follows: X represents the state at time step t, which can also be understood as the context data state at time step t during the execution of the training method by the training device; X represents the user question. , represents the sequence of inference actions performed from time step 1 to time step t-1. In this sequence, the inference action can be understood as the detailed execution process of the inference task. Each inference task may include at least one inference action. This indicates the reasoning action performed at step k, including generating text or invoking a tool; This represents the observations produced after the inference action of the invoked tool is completed; This represents a set of tools that can be invoked, which includes multiple tools. This indicates an indicator function, which is 1 when the condition within the parentheses is met, and 0 otherwise.

[0050] In one specific implementation, the following joint loss function can be used during the training of the target agent based on the target dialogue trajectory data.

[0051] in: D represents the total loss function of the training device during the execution of the training method; D represents the task set of the inference task. Represents the weight coefficient of the i-th reasoning task; Let represent the supervised learning loss for the i-th inference task; The parameters are The target agent in the time step state Generate reasoning action The probability of; The regularization coefficient represents the consistency loss; This represents consistency loss, used to constrain the logical consistency between the reasoning chain and the question answer.

[0052] In this embodiment, to enable the target agent to possess multiple atomic capabilities, the training device employs a multi-task joint optimization method. By introducing weight coefficients for inference tasks, the target agent can learn multiple inference tasks within the same weight space. Furthermore, this is achieved by introducing a consistency loss. To ensure that the reasoning chain generated by the target intelligent agent can logically deduce the answer to the question by the final reasoning action, it is necessary to reduce the occurrence of "illusions" or the disconnect between reasoning and results.

[0053] In some implementations, the training device can extract identity information and corresponding dialogue behavior data from real dialogue data; wherein the identity information is used to characterize the identity role participating in the dialogue in the real dialogue data; the dialogue behavior data is used to characterize the language behavior habits of the corresponding identity during the dialogue process; the real dialogue data is dialogue data that actually occurs in the real world; and the first intelligent agent and the second intelligent agent are constructed based on the identity information and the corresponding dialogue behavior data, respectively.

[0054] In this embodiment, the training device can also be used to extract identity information and corresponding dialogue behavior data from real dialogue data. Real dialogue data can be records of dialogues that actually occur in the real world, typically originating from interactions between users and service personnel in specific transaction scenarios. The training device can parse the real dialogue data to identify the roles and identities of participants in the dialogue, thereby obtaining identity information. Identity information is used to characterize the roles played by different participants in the real dialogue. For example, users, consultants, or transaction handlers. In some embodiments, identity information can be identified by identifying role tags, language direction, contextual features, or role annotations recorded by the system in the dialogue text.

[0055] In this embodiment, based on the acquired identity information, the training device can further extract dialogue behavior data corresponding to the identity information. Dialogue behavior data is used to characterize the language behavior habits of the corresponding identity roles in real dialogues. Dialogue behavior data may include tone patterns, common expressions, structural characteristics of questions or answers, logical preferences in information expression, and response styles for different types of questions. The training device can use natural language processing algorithms to mine the aforementioned dialogue behavior data from real dialogue data, thereby describing the language behavior characteristics of different identity roles in dialogues.

[0056] In this embodiment, the training device can construct a first agent and a second agent based on the acquired identity information and corresponding dialogue behavior data. Specifically, the training device can determine the identity role that the agent needs to simulate based on the identity information, and use the dialogue behavior data to construct a language generation pattern, expression style, and response logic that matches the identity role. For example, for the user role, the first agent can be constructed as a model with common user inquiry behaviors; while for the customer consultation service role, the second agent can be constructed as a model with corresponding identity response characteristics and reasoning tendencies. Through the above construction method, the first agent and the second agent can respectively simulate the real dialogue behaviors of different users and customer consultation service personnel, and generate user questions and answers that are closer to real scenarios in subsequent simulated dialogues.

[0057] Please see Figure 1 In some implementations, the target agent is configured with multiple types of designated reasoning tasks; the training device constructs a reasoning tree based on the reasoning chain of the target dialogue trajectory data; wherein the reasoning tree uses any reasoning task in the reasoning chain as the root node, and adds multiple types of designated reasoning tasks of the target agent; a node score is given for each task node in the reasoning tree; wherein the node score is used to characterize the matching degree of the reasoning task performed at the corresponding task node; the reasoning tree is traversed, and a first target reasoning chain is selected based on the node score; wherein the first target reasoning chain starts from the root node and extends to a leaf node of the reasoning tree, and the question answer obtained by executing the first target reasoning chain is superior to the question answers obtained by other reasoning chains in the reasoning tree.

[0058] In this embodiment, during the training process of the target agent based on the target dialogue trajectory data, the target agent can be pre-set with various types of designated reasoning tasks. These designated reasoning tasks describe different reasoning behaviors that the target agent may perform when handling user questions. For example, designated reasoning tasks may include needs understanding tasks, action planning and tool invocation tasks, proactive dialogue tasks, knowledge reasoning and computation tasks, solution recommendation and product marketing tasks, anthropomorphic response generation tasks, and reflection and self-evolution tasks. Designated reasoning tasks can characterize the diverse reasoning abilities of the target agent in generating question answers. After acquiring the target dialogue trajectory data, the training device can use the reasoning chains contained within it as the basic structure of the reasoning process to further introduce multiple designated reasoning tasks of the target agent into the training process, thereby enabling the target agent to perform reasoning training based on more complex task combinations.

[0059] The needs understanding task is used to identify the explicit needs and potential implicit intentions expressed in user questions. For example, when a user asks, "Can this medicine be used by children?", the needs understanding task can identify it as "concern about the safety of medication for children." Similarly, when a user asks, "I've been taking it for two months and there's still no effect. Is there a problem with the medicine?", the needs understanding task can identify it as "doubt about the expected course and efficacy of the medication." The action planning and tool invocation task is used to determine whether external information sources or internal modules need to be invoked during the reasoning process and to plan the execution order of tool invocations. For example, when the target agent needs to determine the indications, dosage, or contraindications of a drug, action planning can include specific actions such as "invoking the drug database," "matching SKU parameters," and "retrieving medical knowledge." The proactive dialogue task is used to automatically generate supplementary questions when user information is insufficient or requires clarification, proactively guiding the dialogue process. For example, if a user only describes "I've been coughing recently," the proactive dialogue task can generate "Is the cough accompanied by fever or wheezing?" to obtain more complete information. The knowledge reasoning and computation task is used to deduce intermediate conclusions based on medical common sense, knowledge base information, or symptom logic. For example, based on "persistent fever exceeding 40℃," it infers "potential safety risks requiring offline medical attention," or based on "symptoms of runny nose and nasal congestion," it infers "possible allergies or a mild cold." The solution recommendation and product marketing task is used to generate optimal product combinations or service solutions after reasoning leads to user needs. For example, based on a user's description of "nasal congestion and headache," the system can recommend a combination of "anti-allergy medication + saline nasal spray," or provide safer alternatives when the user is concerned about drug side effects. The anthropomorphic response generation task is used to transform structured reasoning content into natural, fluent, and emotionally expressive human language. For example, after arriving at the reasoning result "medication is usable and should be taken after meals," the anthropomorphic response generation task can generate "This medicine is suitable for children; taking it after meals will be milder." The reflection and self-evolution task is used to re-verify the completeness, consistency, and safety of the reasoning process after the reasoning chain is generated, and to correct unreasonable reasoning nodes when necessary. For example, when a recommendation conflicts with a drug contraindication, the reflection task can be modified to "use a safer alternative".

[0060] In this embodiment, the training device can construct a reasoning tree based on the reasoning chain of the target dialogue trajectory data. The reasoning chain includes multiple reasoning tasks, and the training device can use any reasoning task in the reasoning chain as the root node of the reasoning tree. Based on this, the training device can add various types of specified reasoning tasks of the target agent to the reasoning tree to expand the reasoning space. In this way, the reasoning tree can form a multi-level reasoning structure starting from the original reasoning chain and combining it with the specified reasoning tasks possessed by the target agent. This allows the training device to evaluate the performance of different reasoning paths in answering user questions based on the reasoning tree.

[0061] In this embodiment, the training device can assign a node score to each task node in the inference tree. The node score characterizes the degree of matching between the inference task involved in executing the corresponding task node and the target dialogue trajectory data. Specifically, the training device can determine whether a task node has a high degree of fit with the inference chain in the target dialogue trajectory data based on its inference content, contextual relevance, and intermediate inference conclusions derived from that task node, thereby generating a node score. The node score can be used to reflect the rationality and contribution of different inference tasks in the overall inference process, thus providing a basis for subsequently selecting the first target inference chain.

[0062] In this embodiment, the training device can traverse the inference tree and select a first target inference chain based on the node scores of each task node. The first target inference chain can be a inference path from the root node of the inference tree to any leaf node, and the training device can determine the overall quality of the inference path based on the node scores of all task nodes along the inference path. In some embodiments, the training device can calculate the average node score of multiple inference chains in the inference tree and can select the inference chain with the higher average node score as the first target inference chain. Of course, in some embodiments, when the answer to the question corresponding to a certain path is superior to other inference paths in the inference tree in terms of correctness, completeness, or logical consistency, the training device can determine that path as the first target inference chain.

[0063] In one specific implementation, the training device can determine the first target inference chain in the inference tree based on the Monte Carlo Tree Search (MCTS) algorithm.

[0064] This indicates the reasoning action selected during the selection phase of the Monte Carlo tree; , indicating the state at time step The following reasoning action is taken. The average action value can be obtained by dividing the cumulative reward by the number of visits; , representing the exploration constant, is used to balance exploitation and exploration. This represents the prior probability, the probability value directly output by the target agent; , indicating the parent node (time step state) Total number of visits; This indicates the current node (time step state). The following reasoning action is taken. The number of visits to ).

[0065] In this implementation, a Monte Carlo tree algorithm is introduced for end-to-end exploration of the inference chain. During the Monte Carlo tree selection phase, a Predictor + Upper Confidence bound applied to Trees (PUCT) algorithm is employed. The training device can incorporate the known value of the current inference action. and prior probability By number of visits Adjustments are made to favor exploring untried paths in the early stages of reasoning and utilizing high-value paths in the later stages, thereby helping to find a better solution. Of course, this implementation is not limited to using the upper confidence interval algorithm; other forms of UCB (Upper Confidence Bound) variants or search tree algorithms can also be applied to this application.

[0066] In one specific implementation, the training device may select the first target inference chain based on the following scoring method.

[0067] , indicating the selected first target reasoning chain; , represents any one of the reasoning chains in the set of multiple reasoning chains in the reasoning tree; , representing the inference chain obtained by executing the Monte Carlo tree algorithm. Value estimation; , representing the quality score of the reasoning action at step t in the reasoning chain by the Process Reward Model; , representing the total number of steps in the reasoning chain; and , representing the weighting coefficient, is used to balance the importance of global search value and progressive inference quality.

[0068] In this embodiment, the training device can select the first objective reasoning chain based on a comprehensive score. The training device integrates the global search value (representing long-term benefits) of the Monte Carlo tree algorithm and the step-by-step score (representing logical rigor) of the process reward model. This dual verification mechanism ensures that the output question answer not only conforms to the long-term goal (such as solving the user's problem) but also maintains logical correctness at each step of the reasoning, significantly reducing the risk of erroneous responses.

[0069] In one specific implementation, the training device can perform self-game reinforcement of the target agent and the process reward model. Specifically, the self-game objective function is: This can be specifically broken down into adversarial updates between the target agent and the process reward model. Updates of the target agent: Updates to the process reward model: in, , used to represent the model parameters of the target agent; , representing the model parameters of the process reward model; , represents the generation strategy of the target intelligent agent; , representing the learning rate of the target agent; , represents the advantage function at time step t, which measures the quality of reasoning actions relative to the average level; , indicating sample label, =1 indicates that the reasoning action in this step is judged to be of high quality. =0 indicates that the reasoning action in this step is judged to be of low quality; This indicates that the process reward model predicts and infers actions. The probability of high quality; , which represents a preference dataset that includes annotations by human experts or high-quality rule filtering.

[0070] In this embodiment, a self-game framework between the target agent and the process reward model is constructed in the training device. The process reward model minimizes the cross-entropy loss. The goal is to learn to distinguish between high-quality and low-quality inference steps, thereby improving the accuracy of the "judge"; furthermore, the target agent uses the feedback signals (as reward values) provided by the process reward model to update the model parameters through policy gradient algorithms (such as PPO, GRPO). This allows the generation of inference chains that can fool the process reward model. In this way, the training device can optimize the training process without the need for large-scale manual annotation.

[0071] In one specific implementation, the training device can update the model parameters of the target agent based on the following "hybrid dominance function". .

[0072] Among them, the hybrid advantage function Defined as: in, Model parameters used to represent the target agent The direction of the updated gradient; This indicates the target agent's state at time step. Generate reasoning action The probability of; This represents a mixed-dominance signal used to measure the state at a time step. Generate reasoning action The advantages and disadvantages; , used to represent process reward, is a score given by the process reward model to the reasoning logic at step t, so that the target agent not only focuses on the result, but also receives fine-grained feedback at each reasoning step; , used to represent the search value estimate, is the value estimate obtained by backtracking after searching multiple steps backward using the Monte Carlo tree algorithm. It is used to represent the long-term benefits that may be obtained if the current reasoning action is taken, so as to reflect the global planning capability. This is used to represent the value baseline, which is usually predicted by the Critic network to reduce variance; and , is used to represent the weighting coefficient.

[0073] In this embodiment, during the self-game training process, the target agent does not blindly try and fail, but updates itself by minimizing the negative value of the mixed advantage function. By integrating the discriminative power of the process reward model and the exploratory power of the Monte Carlo tree algorithm, the target agent can improve the logicality of reasoning (constrained by the process reward model) and the globality of decision-making (constrained by the Monte Carlo tree algorithm) with less manual annotation. Specifically, This can improve the correction capability of the training device; even if no result has been produced, as long as a certain inference step is accurate, the process reward model will give a positive gradient. Furthermore, in the formula... The introduction of a global perspective for guidance means that gradient updates not only rely on the target agent itself, but also utilize the deep search results of the Monte Carlo tree algorithm as a "teacher" to guide the target agent.

[0074] In some implementations, the training device can filter out a second target inference chain based on the node scores; wherein the second target inference chain starts from the root node and extends to a leaf node of the inference tree, and the question answer derived from the second target inference chain is inferior to the question answer derived from the first target inference chain; the target agent is trained based on the first target inference chain and the second target inference chain, so that the inference chain of the target agent for user questions in the target dialogue trajectory data is closer to the first target inference chain and farther away from the second target inference chain.

[0075] In this embodiment, after the training device determines the first target inference chain based on the inference tree, it can further select a second target inference chain from the inference tree based on node scores. The second target inference chain can be a inference path from the root node to any leaf node of the inference tree. Compared to the first target inference chain, the question answer corresponding to the second target inference chain is inferior in terms of correctness, completeness, or logical consistency. The training device can comprehensively determine the overall quality of each inference chain based on the node scores of all task nodes on the inference path, and use the inference chain with relatively lower overall quality as the second target inference chain to distinguish the quality of the target agent's inference behavior. For example, the inference chain with the lowest average node score is selected from the inference tree as the second target inference chain.

[0076] In this embodiment, the training device can train the target agent based on a first target inference chain and a second target inference chain. Specifically, the first target inference chain is used to characterize the combination of inference steps, the execution order of the specified inference task, and the intermediate inference conclusions of a high-quality inference path in the target dialogue trajectory data. The second target inference chain is used to characterize a low-quality inference path and its inference bias. The training device can compare the inference chain output by the target agent during the training process on the target dialogue trajectory data with the first and second target inference chains, prompting the target agent to generate inference chains that are closer to the high-quality inference process represented by the first target inference chain and further away from the low-quality inference process represented by the second target inference chain.

[0077] In this embodiment, the training device can utilize the differences between the first target reasoning chain and the second target reasoning chain to provide the target agent with a clear direction for reasoning optimization. This enables the target agent to gradually learn reasoning content and the execution order of specified reasoning tasks that are more in line with high-quality reasoning performance when facing user questions in the target dialogue trajectory data, thereby improving the reasoning accuracy of the target agent.

[0078] In some implementations, the training device can train a specified agent based on the first target inference chain and the second target inference chain; wherein the parameter size of the specified agent is larger than the parameter size of the target agent; and distillation learning is performed on the target agent based on the specified agent.

[0079] In this embodiment, after the training device determines the first target inference chain and the second target inference chain, it can further train the specified agent based on these two inference chains. The parameter size of the specified agent is larger than that of the target agent, allowing it to learn the differential features of complex inference paths on a larger model scale. The training device can use the first target inference chain as an example of a high-quality inference path and the second target inference chain as an example of a low-quality inference path, enabling the specified agent to learn the significant differences between the quality of inference paths.

[0080] In this embodiment, the designated agent can learn the structural differences, inference step differences, and quality of intermediate inference information between the first target inference chain and the second target inference chain, thereby forming a preference for high-quality inference chains and an ability to avoid low-quality inference chains in its internal parameters.

[0081] In this embodiment, the training device can perform distillation learning on the target agent based on a specified agent. Specifically, the training device can transfer the reasoning preferences learned by the specified agent to the target agent, enabling the target agent to retain the high-quality reasoning ability preserved in the specified agent even with a limited parameter scale. The training device can guide the target agent to generate reasoning chains closer to the first target reasoning chain and reduce the tendency to move closer to the second target reasoning chain. In this way, the target agent can still achieve good reasoning accuracy while maintaining a small model size.

[0082] Please see Figure 2 In some implementations, the training device can execute specified inference tasks to obtain inference information and task results corresponding to the specified inference tasks. The input to each specified inference task includes at least task definition data, real dialogue data, and task execution data. The task definition data describes the function implemented by the corresponding specified inference task. The task execution data is the sample data required to execute the corresponding specified inference task, and the task execution data includes at least the target dialogue trajectory data. A task evaluation score is generated for each specified inference task's inference information and task result. The task evaluation score represents the degree of matching between the inference information and task result of the corresponding specified inference task and the real dialogue data. Combining the inference information, task results, and task evaluation scores of the various types of specified inference tasks, comprehensive inference processing is performed to obtain target training samples. The target training samples include agent function definitions, training execution data, and sample inference chains. The agent function definitions describe the function implemented by the target agent. The training execution data is transaction sample data for training the target agent. The sample inference chain includes specified inference tasks corresponding to the target training samples. The target agent is trained using the target training samples.

[0083] In this embodiment, during the training process of the target agent based on the target dialogue trajectory data, the training device can execute various types of specified reasoning tasks possessed by the target agent to obtain corresponding reasoning information and task results. The input to the specified reasoning task may include task definition data, real dialogue data, and task execution data. The task definition data describes the functional objectives achieved by the specified reasoning task, the real dialogue data serves as a reference for task execution, and the task execution data consists of sample data required to execute the specified reasoning task, including at least the target dialogue trajectory data. The training device can trigger the execution process of each specified reasoning task based on the above input data to generate corresponding reasoning information and task results from different reasoning perspectives. In some embodiments, the task execution data may also include: user questions, contextual communication data surrounding the user questions during the simulated dialogue between the first and second agents, and some basic transaction data required for the specified reasoning task. This allows for more comprehensive information to be obtained during the execution of the specified reasoning task.

[0084] In this embodiment, the training device can generate a task evaluation score for the reasoning information and task results of each specified reasoning task. The task evaluation score characterizes the degree of matching between the execution process of the specified reasoning task and the real dialogue data. The training device can quantitatively evaluate the output of each specified reasoning task by combining the logical consistency of task execution, the fit between the reasoning content and the real dialogue data, and the reasonableness of the task results in the transaction scenario. The task evaluation score can reflect the quality performance of different specified reasoning tasks on specific samples, providing an evaluation basis for subsequent comprehensive reasoning processing.

[0085] In this embodiment, the training device can combine reasoning information, task results, and task evaluation scores from multiple types of specified reasoning tasks to perform comprehensive reasoning processing to construct target training samples. For example, the training device can pre-build a reasoning module, which can be a machine learning model trained on a large language model. The training device can call this reasoning module to perform comprehensive reasoning processing to obtain target training samples. Target training samples may include agent function definitions, training execution data, and sample reasoning chains. Agent function definitions describe the functional goals that the target agent needs to achieve in a specific scenario. Training execution data consists of transaction sample data required to train the target agent. Sample reasoning chains include specified reasoning tasks corresponding to the target training samples and their execution order. The training device can integrate reasoning results from multiple specified reasoning tasks into training samples through comprehensive reasoning processing to characterize the reasoning behavior required by the target agent. In this embodiment, training execution data may include: user questions, contextual communication data surrounding the user questions during simulated dialogue between the first and second agents, etc.

[0086] In this embodiment, the training device can train the target agent using target training samples. Specifically, the training device can guide the target agent to learn corresponding reasoning abilities and task execution rules based on the agent function definitions, training execution data, and sample reasoning chains in the target training samples. In this way, the target agent can form a more stable reasoning process and dialogue generation ability by integrating the reasoning results of multiple specified reasoning tasks.

[0087] In some implementations, the training device can obtain a first feedback signal for the target agent to execute training on the target training sample according to a first reward function, a second feedback signal for the target agent to execute training on the target training sample according to a second reward function, and a third feedback signal for the target agent to execute training on the target training sample according to a third reward function; wherein, the first feedback signal is used to characterize the degree of matching between the current question answer output by the target agent corresponding to the target training sample and the specified standard question answer in the target training sample; the second feedback signal is used to characterize the degree of matching between the inference chain of the target agent corresponding to the target training sample and the sample inference chain; the third feedback signal is used to characterize the degree of matching between the current question answer of the target agent and the inference chain corresponding to the target training sample; and the target agent is adjusted according to the first feedback signal, the second feedback signal, and the third feedback signal.

[0088] In this embodiment, when training the target agent using target training samples, the training device can perform reinforcement adjustments on the target agent based on multi-granularity reward signals. Specifically, the training device can first obtain a first feedback signal based on a first reward function to measure the degree of matching between the current question answer output by the target agent and the specified standard question answer in the target training samples. The first reward function can be a Response reward for the question answer, and its calculation method is as follows: in, This represents the target agent's answer to the current question output from the target training samples. This represents the answers to pre-given standard questions in the target training samples. This indicates the similarity between the two. Similarity It can be calculated by the training device based on edit distance or semantic similarity, for example: in, for and Edit distance, and These represent the lengths of the current question's answer and the standard question's answer, respectively. The training device uses these lengths to obtain the first feedback signal, which directly drives the target agent to improve the correctness and consistency of its answers.

[0089] In this embodiment, the training device can also obtain a second feedback signal based on the second reward function to measure the degree of matching between the inference chain generated by the target agent and the sample inference chain. The second reward function can be an Action reward, used to evaluate the specified inference task in the inference chain, and its form is, for example: in, This represents the sequence of specified inference tasks given by the target agent in the current inference chain. This represents the sequence of preferred inference tasks that have been manually verified in the target training samples, corresponding to the preferred inference path embodied in the sample inference chain; This indicates that the sequence of the currently specified inference task matches the sequence of preferred inference actions only in some nodes or in order. The training device obtains a second feedback signal through this second reward function, enabling the target agent to gradually learn task selection and execution order that are closer to the sample inference chain.

[0090] In this embodiment, the training device can also obtain a third feedback signal based on the third reward function to measure the degree of matching between the target agent's current question answer and the corresponding inference chain, and to constrain the normalization of the output structure. The third reward function can be a format reward, for example, as follows: in, The output of the target agent (including the reasoning chain and the question answer) is represented by the training device. The training device can check the format correctness based on the output structure required by the sample reasoning chain. For example, it may only allow the thought segment and the answer segment to be output correctly and in the correct order once, to avoid skipping reasoning, incorrect placement of reasoning segments, or inconsistencies between reasoning and answer. Based on this, the training device obtains a third feedback signal to encourage the target agent to maintain structural consistency between the reasoning process and the answer.

[0091] In this embodiment, the training device can also fuse the three types of feedback signals to train the target agent as a whole. For example, the training device can use a general reward function to weight and combine the three types of rewards: in, and For weight parameters, , and These correspond to the first reward function, the second reward function, and the third reward function, respectively. The training device can adjust the parameters of the target agent based on this combined reward, using the first, second, and third feedback signals in conjunction, thereby simultaneously enhancing the target agent's correctness of responses, matching of inference chains, and consistency between inference and responses.

[0092] Please see Figure 5 The embodiments of this application also provide a training device for an intelligent agent. The training device includes: an acquisition module, a determination module, and a training module.

[0093] An acquisition module is used to acquire dialogue trajectory data during a simulated dialogue between a first agent and a second agent; wherein the first agent and the second agent respectively imitate different identities and roles; the dialogue trajectory data includes at least a user question generated by the first agent, a reasoning chain of the second agent corresponding to the user question, and a question answer generated by the second agent based on the reasoning chain; the reasoning chain includes multiple reasoning tasks, and the multiple reasoning tasks are used to characterize the reasoning process of the second agent in response to the user question to generate a question answer.

[0094] The determination module is used to determine target dialogue trajectory data based on the quality score of the dialogue trajectory data; wherein, the quality score is used to represent the degree of fit between user questions and answers in the dialogue trajectory data; the degree of fit between user questions and answers in the target dialogue trajectory data is higher than the degree of fit between user questions and answers in non-target dialogue trajectory data.

[0095] The training module is used to train the target agent based on the target dialogue trajectory data.

[0096] In this embodiment, the functions and effects of the training device for the intelligent agent can be explained in comparison with the aforementioned embodiments, and will not be repeated here.

[0097] Please see Figure 6 This application also provides a computer device comprising: a memory and a processor, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the method described above.

[0098] The memory, processor, and communication interface in the computer device can communicate with each other via the system bus and network communication.

[0099] In this embodiment, the functions and effects implemented by the computer device can be explained by referring to the foregoing embodiments, and will not be repeated here.

[0100] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, causes the processor to implement the method as described above.

[0101] The functions and effects achieved in this embodiment can be explained by referring to other embodiments, and will not be repeated here.

[0102] This application also provides a computer program product containing instructions, including a computer program / instructions that, when executed by a processor, implement the method as described above.

[0103] The functions and effects achieved in this embodiment can be explained by referring to other embodiments, and will not be repeated here.

[0104] It is understood that the specific examples in this document are only intended to help those skilled in the art better understand the embodiments of this application, and are not intended to limit the scope of the invention.

[0105] It is understood that in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0106] It is understood that the various implementation methods described in this application can be implemented individually or in combination, and the implementation methods in this application are not limited in this respect.

[0107] Unless otherwise stated, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items. The singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0108] It is understood that the processor in the embodiments of this application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiments can be completed by the integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0109] It is understood that the memory in the embodiments of this application may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Specifically, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM). It should be noted that the memory in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0110] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the aforementioned method implementations, and will not be repeated here.

[0112] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0114] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0115] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The above description is merely a specific embodiment of this application, but the scope of protection of this invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this invention should be determined by the scope of the claims.

Claims

1. A method for training an intelligent agent, characterized in that, include: The dialogue trajectory data is obtained during a simulated dialogue between a first agent and a second agent; wherein the first agent and the second agent respectively imitate different identities and roles; the dialogue trajectory data includes at least a user question generated by the first agent, a reasoning chain of the second agent corresponding to the user question, and a question answer generated by the second agent based on the reasoning chain; the reasoning chain includes multiple reasoning tasks, and the multiple reasoning tasks are used to characterize the reasoning process of the second agent in response to the user question to generate a question answer; Target dialogue trajectory data is determined based on the quality score of the dialogue trajectory data; wherein, the quality score is used to represent the degree of fit between user questions and answers in the dialogue trajectory data; the degree of fit between user questions and answers in the target dialogue trajectory data is higher than the degree of fit between user questions and answers in non-target dialogue trajectory data; The target agent is trained based on the target dialogue trajectory data.

2. The method according to claim 1, characterized in that, The method further includes: From real-world dialogue data, identity information and corresponding dialogue behavior data are extracted; wherein, the identity information is used to represent the identity role participating in the dialogue in the real-world dialogue data; the dialogue behavior data is used to represent the language behavior habits of the corresponding identity during the dialogue process; the real-world dialogue data refers to dialogue data that actually occurs in the real world. The first intelligent agent and the second intelligent agent are constructed based on the identity information and the corresponding dialogue behavior data, respectively.

3. The method according to claim 1, characterized in that, The target intelligent agent is configured with multiple types of designated reasoning tasks; Training a target agent based on target dialogue trajectory data includes: Based on the inference chain of the target dialogue trajectory data, an inference tree is constructed; wherein, the inference tree takes any inference task in the inference chain as the root node and adds various types of specified inference tasks of the target agent; A node score is given for each task node in the inference tree; wherein, the node score is used to characterize the degree of matching of the inference task performed on the corresponding task node; Traverse the inference tree and select a first target inference chain based on the node scores; wherein the first target inference chain starts from the root node and extends to a leaf node of the inference tree, and the question answer obtained by the first target inference chain is superior to the question answer obtained by other inference chains in the inference tree.

4. The method according to claim 3, characterized in that, The method further includes: A second target inference chain is selected based on the node scores; wherein the second target inference chain starts from the root node and extends to a leaf node of the inference tree, and the question answer obtained by the second target inference chain is inferior to the question answer obtained by the first target inference chain. The target agent is trained based on the first target inference chain and the second target inference chain, so that the inference chain of the target agent for user questions in the target dialogue trajectory data is closer to the first target inference chain and farther away from the second target inference chain.

5. The method according to claim 4, characterized in that, The method further includes: A designated agent is trained based on the first target inference chain and the second target inference chain; wherein the parameter size of the designated agent is larger than the parameter size of the target agent; Distillation learning is performed on the target agent based on the specified agent.

6. The method according to claim 1, characterized in that, The target intelligent agent is configured with multiple types of designated reasoning tasks; Training a target agent based on target dialogue trajectory data includes: Each specified reasoning task is executed to obtain the reasoning information and task results for the corresponding specified reasoning task; wherein, the input of the specified reasoning task includes at least task definition data, real dialogue data, and task execution data; the task definition data is used to describe the function implemented by the corresponding specified reasoning task; the task execution data is the sample data required to execute the corresponding specified reasoning task, and the task execution data includes at least the target dialogue trajectory data; For each specified reasoning task, a task evaluation score is generated based on the reasoning information and task results. The task evaluation score represents the degree of matching between the reasoning information and task results of the corresponding specified reasoning task and the real dialogue data. By combining the reasoning information, task results, and task evaluation scores of the various types of specified reasoning tasks, a comprehensive reasoning process is performed to obtain a target training sample. The target training sample includes an agent function definition, training execution data, and a sample reasoning chain. The agent function definition describes the functions implemented by the target agent. The training execution data consists of transaction sample data used to train the target agent. The sample reasoning chain includes specified reasoning tasks corresponding to the target training sample. The target agent is trained using the target training samples.

7. The method according to claim 6, characterized in that, Training the target agent using the target training samples includes: The system obtains a first feedback signal for the target agent to train the target training sample according to a first reward function, a second feedback signal for the target agent to train the target training sample according to a second reward function, and a third feedback signal for the target agent to train the target training sample according to a third reward function; wherein, the first feedback signal is used to characterize the degree of matching between the current question answer output by the target agent corresponding to the target training sample and the specified standard question answer in the target training sample; the second feedback signal is used to characterize the degree of matching between the inference chain of the target agent corresponding to the target training sample and the sample inference chain; and the third feedback signal is used to characterize the degree of matching between the current question answer of the target agent and the inference chain corresponding to the target training sample. The target intelligent agent is adjusted based on the first feedback signal, the second feedback signal, and the third feedback signal.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, causes the processor to implement the method as described in any one of claims 1 to 7.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 7.

Images