Intelligent agent training methods, data processing methods, and question answering methods

Abstract

This specification provides an agent training method, a data processing method, and a question-answering method. The agent training method includes: training an initial agent using multi-round reference data and corresponding reference results to obtain an updated agent; obtaining multi-round exploration data and corresponding exploration results based on the updated agent; determining positive and negative results based on the annotation information of the multi-round reference results and multi-round exploration results, wherein the guidance results included in the reference results and / or exploration results are positive results, and the guidance results are used to guide the interacting party to output data containing atomic facts; training the updated agent based on the target data, positive results, and negative results to obtain a target agent; and enabling the target agent to learn high-value clarification (guidance) strategies through guided responses, thereby gaining the ability to proactively guide dialogue, significantly reducing ineffective communication, solving the inefficiency problem caused by missing information, and greatly improving service efficiency and accuracy.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to a method for training an intelligent agent. One or more embodiments of this specification also relate to data processing methods, question-answering methods, computing devices, computer-readable storage media, and computer program products. Background Technology

[0002] In highly knowledge-intensive fields such as law and medicine, expert consultation services are crucial. However, the feasibility of such services has always been challenged due to the scarcity of professional talent and knowledge barriers. Taking legal consultation as an example, non-professional users often lack industry background knowledge and find it difficult to accurately describe their problems, making it difficult for experts to provide timely and targeted answers. This results in an inefficient consultation process and lengthy communication.

[0003] Even with the introduction of artificial intelligence to improve consultation efficiency, most AI models remain at the level of knowledge retrieval and basic question answering, and cannot deeply understand the atomic facts corresponding to the consultation event in order to give an accurate response; that is, in real-world scenarios, user input is often vague, and AI models have difficulty effectively understanding the facts, often giving irrelevant or templated responses. Summary of the Invention

[0004] In view of the above, embodiments of this specification provide an agent training method. One or more embodiments of this specification also relate to data processing methods, question-answering methods, computing devices, computer-readable storage media, and computer program products, in order to solve the technical defects existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, an agent training method is provided, comprising:

[0006] The initial agent is trained using multiple rounds of reference data and corresponding reference results to obtain the updated agent;

[0007] Based on the updated agent, multiple rounds of exploration data and corresponding exploration results are obtained, wherein the first round of exploration data is the same as the first round of reference data;

[0008] Based on the annotation information of the multi-round reference results and the multi-round exploration results, positive results and negative results are determined. Among them, the guidance results included in the reference results and / or exploration results are positive results. The guidance results are used to guide the interaction party to output data containing atomic facts.

[0009] The updated agent is trained based on the target data, the positive results, and the negative results to obtain the target agent, wherein the target data is reference data or exploration data.

[0010] According to a second aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0011] Initial data is input into the target agent to obtain the initial result output by the target agent, wherein the target agent is obtained according to the above-described agent training method;

[0012] In response to the response data corresponding to the initial result, the response data is determined as the initial data, and the step of inputting the initial data into the target agent is continued until the stopping condition is met.

[0013] According to a third aspect of the embodiments of this specification, a question-and-answer method is provided, including:

[0014] An initial question is input into the target agent, and an initial response is obtained from the target agent, wherein the target agent is obtained according to the agent training method described above;

[0015] In response to the response question corresponding to the initial response, the response question is determined as the initial question, and the step of inputting the initial question into the target agent is continued until the stopping condition is met.

[0016] According to a fourth aspect of the embodiments of this specification, a question-answering method is provided, applied to a question-answering platform, wherein a target intelligent agent is deployed on the question-answering platform, comprising:

[0017] In response to an inquiry request sent by the client, obtain the initial question carried in the inquiry request;

[0018] The initial question is input into the target agent, the initial response output by the target agent is obtained, and the initial response is returned to the client, wherein the target agent is obtained according to the above-described agent training method;

[0019] In response to a follow-up question sent by the client corresponding to the initial response, the response question corresponding to the initial response is obtained, the response question is identified as the initial question, and the step of inputting the initial question into the target agent is continued until the stopping condition is met.

[0020] According to a fifth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0021] Memory and processor;

[0022] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the question-answering method and the agent training method described above.

[0023] According to a sixth aspect of the embodiments of this specification, an electronic device is provided, comprising:

[0024] A memory and a processor, the memory and the processor being connected via a bus;

[0025] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, they implement the steps of the above-mentioned question-answering method and agent training method.

[0026] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions, which, when executed by a processor, implement the steps of the above-described question-answering method and agent training method.

[0027] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described question-answering method and agent training method.

[0028] The agent training method provided in one embodiment of this specification, when training a target agent using data containing positive and negative results, allows the positive and negative results to directly influence the agent's policy updates, making the target agent more inclined to generate policies that produce positive results and avoid those that produce negative results. These positive and negative results are determined based on the annotation information of the reference data corresponding to the reference results and the exploration data corresponding to the exploration results. When the positive results include guidance results, and these guidance results are used to guide the interacting party to output data containing atomic facts, the target agent can learn high-value clarification (guidance) strategies, thereby overcoming the limitations of shallow knowledge retrieval in traditional methods and gaining the ability to proactively guide. Therefore, when applying the target agent, it can generate accurate and guiding initial results from fuzzy, information-deficient initial data, significantly reducing ineffective interactions and saving interaction resources. Based on the target agent's ability to achieve a process from fuzzy data to accurate solutions, it solves the inefficiency problem caused by information deficiencies, greatly improving service efficiency and accuracy. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of a scenario for an agent training method provided in one embodiment of this specification;

[0030] Figure 2 This is a flowchart of an agent training method provided in one embodiment of this specification;

[0031] Figure 3 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0032] Figure 4 This is a flowchart illustrating a question-and-answer method provided in one embodiment of this specification;

[0033] Figure 5 This is a flowchart illustrating a question-and-answer method applied to a question-and-answer platform, as provided in one embodiment of this specification.

[0034] Figure 6 This is a flowchart illustrating the processing procedure of an agent training method provided in one embodiment of this specification.

[0035] Figure 7 This is a schematic diagram of the structure of an intelligent agent training device provided in one embodiment of this specification;

[0036] Figure 8 This is a schematic diagram of the structure of a data processing device provided in one embodiment of this specification;

[0037] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification;

[0038] Figure 10 This is a structural block diagram of an electronic device provided in one embodiment of this specification. Detailed Implementation

[0039] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0040] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0041] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0042] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0043] In one or more embodiments of this specification, a large model refers to a deep learning model with a large number of model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even tens of trillions of model parameters. A large model can also be called a foundation model. It is pre-trained using large-scale unlabeled corpora to produce a pre-trained model with hundreds of millions of parameters. Such models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0044] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as natural language processing tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0045] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0046] Behavioral cloning: a supervised learning method that trains agents by imitating actions demonstrated by experts, often used for learning the basic capabilities of conversational agents.

[0047] DPO: Direct Preference Optimization, is an algorithm that improves the model's preference inheritance and generalization ability by comparing and learning from pairs of preferred and undesirable actions.

[0048] Stepwise Preference Attribution Optimization (SPAO) is a method for fine-grained analysis and attribution of the effectiveness of clarification actions at each step of a multi-turn dialogue, driving dialogue strategy optimization accordingly. The agent's "clarification action" is essentially a guiding action; its core purpose is not to directly answer the user's question, but to proactively guide the user to provide more accurate key facts and information, thereby gradually eliminating ambiguity and uncertainty in the user's question and ultimately achieving precise clarification of the specific event.

[0049] Exploration Trajectory refers to the dialogue sequence generated by an agent during autonomous interaction with the environment and users, used to discover novel or efficient clarifying behaviors in non-expert demonstration paths.

[0050] Domain Event Schema refers to the abstraction of factual information into several higher-order event types (Schema tags) in a specific domain, which organizes and expresses complex knowledge in a structured way, and guides intelligent agents to achieve process-oriented, multi-step reasoning and clarification.

[0051] In recent years, large language models have made significant breakthroughs in knowledge acquisition and multi-turn dialogue, demonstrating enormous potential to upgrade expert consultation services. However, in real-world scenarios, user input is often ambiguous, making it difficult for large language models to effectively clarify it, often resulting in irrelevant or templated responses.

[0052] In the embodiments described in this specification, a new generation of intelligent agents with autonomous exploration and clarification capabilities are combined to enhance the model's multi-round interaction and decision-making capabilities in complex expert consultation tasks. This solves the technical bottlenecks of existing intelligent agents, which are unable to proactively and flexibly clarify user ambiguities and cannot efficiently absorb and generalize key clarification strategies.

[0053] To address the aforementioned technical problems, this specification provides a question-and-answer method. One or more embodiments of this specification also relate to a question-and-answer device, an agent training method, an agent training device, a computing device, an electronic device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0054] See Figure 1 , Figure 1 A schematic diagram of a scenario illustrating an agent training method according to an embodiment of this specification is shown.

[0055] Specifically, the agent training method is applied to an agent training system, which includes an edge device 102 and a server 104. The edge device 102 is used to send reference data to the server 104.

[0056] The server 104 obtains reference results corresponding to the reference data, and trains an initial agent using multiple rounds of reference data and corresponding reference results to obtain an updated agent. Based on the updated agent, it obtains multiple rounds of exploration data and corresponding exploration results, wherein the first round of exploration data is the same as the first round of reference data. Based on the annotation information of the multiple rounds of reference results and the multiple rounds of exploration results, it determines positive and negative results, wherein the guidance results included in the reference results and / or exploration results are positive results, used to guide the interacting party to output data containing atomic facts. Based on the target data, the positive results, and the negative results, it trains the updated agent to obtain a target agent, wherein the target data is either reference data or exploration data. When a target agent is obtained through training, it can be deployed on the edge device 102, or the calling information of the target agent can be sent to the edge device 102; this is not limited here.

[0057] The edge device 102 may include a browser, an app (application), or a web application such as an H5 (Hypertext Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. The edge device can be developed based on a software development kit (SDK) provided by the server, such as a real-time communication (RTC) SDK. The edge device can be deployed in an electronic device and depends on the device's operation or certain apps within the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured in the electronic device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0058] Server 104 can be understood as a server providing various services, including physical servers and cloud servers. Examples include servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It's important to note that Server 104 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. Server 104 can also be a server in a distributed system, or a server integrated with blockchain. Server 104 can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0059] See Figure 2 , Figure 2 A flowchart of an agent training method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0060] Step 202: Train the initial agent using multiple rounds of reference data and the corresponding reference results to obtain the updated agent.

[0061] In this context, data can be understood as the input and environmental information provided by the interaction method during the interaction process; results can be understood as the output returned to the interaction party after processing and analyzing the data; in a dialogue scenario, data can be understood as the question input by the interaction party, and results can be understood as the response returned after analyzing the input question; in a shopping scenario, data can be understood as the search terms input by the interaction party, and results can be understood as the corresponding understanding response and related recommended products.

[0062] In fact, a reference trajectory can be constructed based on multiple rounds of reference data and corresponding reference results. In a dialogue scenario, a reference trajectory can be understood as a reference dialogue paradigm that includes domain expert knowledge, and includes multiple rounds of reference questions (i.e., reference data) and corresponding reference responses (i.e., reference results).

[0063] Specifically, when the initial agent is trained in a supervised manner using a reference trajectory (i.e., expert demonstration), the updated agent can better understand the contextual semantics and reasoning results, and improve its grasp, induction, and multi-stage policy planning capabilities of domain knowledge.

[0064] In one or more embodiments of this specification, to improve the efficiency of reference trajectory construction and save human resources, preset fact labels can be used to abstract the reference data, thereby combining the fact labels obtained from mapping with the original reference data to generate a reference result. Specific implementation methods are described below:

[0065] Before obtaining the updated agent by training the initial agent using multiple rounds of reference data and corresponding reference results, the process further includes:

[0066] Determine reference data and the fact labels corresponding to the reference data, wherein the reference data is the reference data of the first round in the multi-round reference data, the multi-round reference data corresponds to a domain event containing one or more atomic facts, and the fact label is the mapping label corresponding to the atomic fact;

[0067] Based on the reference data and the fact tags, a reference result corresponding to the reference data is generated, wherein the reference result is the guidance result or the solution result;

[0068] In response to the response reference data corresponding to the reference result, the reference data is updated to the response reference data. The process is iterated through rounds by performing the steps of determining the reference data and the fact labels corresponding to the reference data to obtain the multi-round reference data and the corresponding reference result.

[0069] In this context, reference data can be understood as questions or statements output by the interacting party during the interaction process. This reference data may contain one or more atomic facts corresponding to domain events. Domain events can be understood as events that occur in a specific domain, and atomic facts can be understood as factual information that constitutes a domain event. Reference results can be understood as standard clarifying or guiding questions generated based on domain event patterns during the interaction process, or as answers to reference data. Fact labels can be understood as predefined semantic labels, which are usually obtained by abstracting factual information into higher-order event types (such as fact labels such as <amount>, <compensation>, <traffic accident>, etc.), and are used to structurally represent the atomic facts contained in the reference data.

[0070] The guiding result can be understood as a follow-up response aimed at guiding the interacting party to provide more factual information, i.e., a clarification action; while the answer result can be understood as a response that directly gives the final answer or suggestion; the response reference data can be understood as the interacting party's response to the reference result, which can be used as a reference question for the next round; in the case of round iteration, the conditions for the end of the round iteration can be that the number of dialogue rounds reaches the upper limit, the agent gives a final answer, or the interacting party actively ends the dialogue, etc., which are not limited here.

[0071] Specifically, starting from the first round, for each round's reference data, combined with its associated fact tags, a corresponding reference result is automatically generated. This reference result can guide the interacting party to clarify the issue, or it can be a direct answer to the reference data. Upon receiving a response from the interacting party—that is, upon receiving the reference question corresponding to the response reference result—the reference question is updated to the response reference question, and the next round of the loop begins. This process continues until the condition for the end of the round iteration is met, ultimately outputting a complete reference trajectory containing multiple rounds of high-quality data pairs (reference data - reference result).

[0072] For example, in a consultation scenario, a user might provide a reference question (i.e., reference data) such as, "I was in a car accident and was asked to pay compensation. Is this reasonable?" Using domain event patterns summarized by domain experts (a predefined labeling system; the labeling system for domain event patterns is flexible and can be adapted to different task scenarios), the atomic facts contained in this reference question are normalized and mapped, resulting in fact labels such as [<traffic accident>, <compensation>]. Based on these fact labels and the reference question, a large model can be automatically invoked to systematically generate a reference response (i.e., a reference result) according to the reasoning chain of the domain event pattern. This reference response should ensure full coverage of the atomic facts of the domain event, effectively reflecting the expert's decision-making logic. Specifically, when the full range of atomic facts of the domain event is not yet understood, the generated reference response can be a guiding question (i.e., a guiding result). This guiding question instructs the expert on what kind of questions to ask to elicit the relevant atomic facts. For example, the generated reference response could be guiding questions such as "What are the specific details of the car accident?" or "How much compensation is the other party demanding?", specifically guiding the understanding of the detailed circumstances of the traffic accident and the specific amount of compensation, etc.

[0073] The agent training method provided in the embodiments of this specification combines atomic facts in the reference question to determine their corresponding fact labels, thereby transforming complex and cumbersome user statements into higher-order demand attributes, building a structured knowledge base, and automatically generating reference responses based on fact labels and reference questions. This can guide the interacting party to output more atomic facts related to domain events, enabling the reference responses to broadly cover various potential questions and achieve a high coverage clarification effect.

[0074] In one or more embodiments of this specification, when determining the fact label corresponding to a reference problem, the domain event corresponding to the reference problem is first determined, and then the fact label corresponding to the reference problem is determined from the set of labels corresponding to the domain events. Specific implementation methods are described below:

[0075] The determination of reference data and the corresponding fact labels includes:

[0076] Determine the reference data and the corresponding domain events;

[0077] Determine the fact label corresponding to the reference data from the label set corresponding to the domain event.

[0078] The tag set can be understood as a library of event type tags (such as <amount>, <time>) that are predefined by domain experts to describe events that may be included in a certain domain; the fact tags can be understood as specific semantic tags that are selected from the tag set and associated with the reference question.

[0079] Specifically, determine the domain of the reference data (such as "legal consultation", "medical consultation", "work injury compensation", etc.), determine the tag set corresponding to the domain of the event, and match the tags corresponding to the factual information contained in the reference data from the tag set, and use them as the fact tags corresponding to the reference data.

[0080] For example, taking events in the field of "workers' compensation claims" as an example, its tag set can include semantic tags such as <incident time>, <incident location>, <injured body part>, and <medical expenses>. In the case of a reference question like "I was injured on my way to work, I fell and broke my arm, can I claim compensation?", it can be determined that this reference question belongs to the field of "workers' compensation claims". Then, the semantic tag <injured body part> is precisely matched from the tag set corresponding to "workers' compensation claims", and <injured body part> is determined as the fact tag corresponding to the aforementioned reference question.

[0081] The agent training method provided in the embodiments of this specification can accurately determine the corresponding semantic label for the reference problem from the label set corresponding to the domain event by determining the domain event corresponding to the reference problem, so as to establish the association between the reference problem and structured knowledge.

[0082] Step 204: Based on the updated agent, obtain multiple rounds of exploration data and corresponding exploration results, wherein the first round of exploration data is the same as the first round of reference data.

[0083] In fact, an exploration trajectory can be constructed based on multiple rounds of exploration data and the corresponding exploration results. This exploration trajectory can be understood as the dialogue record generated by the updating agent in actual interaction with the interacting party, including the exploration data output by the interacting party in multiple rounds and the exploration results generated by the updating agent.

[0084] The interaction party can be understood as the object that interacts with the intelligent agent, such as a user or the environment. In the embodiments of this specification, the interaction party can be a real user or a simulated user who is having a dialogue with the intelligent agent.

[0085] Specifically, the reference trajectory and the exploration trajectory start from the same point, that is, based on the same exploration data as the first round of reference data, they interact with the updated agent. By collecting the interaction data during the interaction process, an exploration trajectory corresponding to the reference trajectory is obtained, which includes multiple rounds of exploration data and the corresponding exploration results.

[0086] In one or more embodiments of this specification, when an updated agent is obtained by training an initial agent using a reference trajectory, multi-turn dialogue simulation and data collection are performed by updating the interaction between the updated agent and the interacting party, thereby obtaining multi-turn exploration data and corresponding exploration results. Specific implementation methods are described below:

[0087] The step of obtaining multi-round exploration data and corresponding exploration results based on the updated agent includes:

[0088] The exploration data is input into the updating agent to obtain the exploration results output by the updating agent, wherein the exploration data is the exploration data of the first round in the multi-round exploration data, and the multi-round exploration data corresponds to a domain event containing one or more atomic facts;

[0089] In response to the response exploration data output by the interactive party corresponding to the exploration result, the exploration data is updated to the response exploration data. The process is repeated round by round by executing the step of inputting the exploration data into the updating agent to obtain the multi-round exploration data and the corresponding exploration results.

[0090] Here, the updated agent can be understood as an agent that has undergone initial training and possesses basic dialogue capabilities but has not yet been fully optimized; exploration data can be understood as natural language questions actively raised or responded to by the interacting party during the interaction process; exploration results can be understood as the response content generated by the updated agent in response to the explorer's questions; and response exploration data can be understood as the follow-up questions or feedback raised by the interacting party based on the updated agent's response, which drive the next round of interaction.

[0091] Specifically, starting from the first round, the exploration data proposed by the interacting party is input into the updating agent, and the exploration results provided by the updating agent are obtained. The interacting party will then provide response exploration data based on these results. Therefore, this response exploration data can be used as the exploration data for the next round, and is input into the updating agent again to obtain the corresponding exploration results. This cycle continues until the condition for the end of the round iteration is met. The exploration data and the exploration results of the updating agent in multiple rounds throughout the entire interaction process are recorded, resulting in multiple rounds of exploration data and corresponding exploration results, forming a complete exploration trajectory.

[0092] It should be noted that the first round of exploration questions and the first round of reference questions need to be consistent, that is, the reference trajectory and the exploration trajectory have the same starting point, and the reference trajectory and the exploration trajectory correspond to the same domain event, so that the reference trajectory and the exploration trajectory can be effectively compared in the future.

[0093] The agent training method provided in the embodiments of this specification can collect and update the agent's behavioral data in real interactive environments by updating the interaction between the agent and the interacting party. By recording its responses in different contexts, it provides rich and realistic data for subsequent preference learning and policy optimization.

[0094] Step 206: Based on the annotation information of the multi-round reference results and the multi-round exploration results, determine the positive results and negative results, wherein the guidance results included in the reference results and / or exploration results are positive results, and the guidance results are used to guide the interactive party to output data containing atomic facts.

[0095] The annotation information can be understood as classifying each result (including reference results and exploration results) and obtaining classification labels, which are used to distinguish whether the result is effective, accurate or beneficial; the target agent can be understood as the final model obtained after multiple rounds of optimization, whose guidance and problem-solving capabilities are superior to the initial agent.

[0096] In practical applications, taking a medical consultation scenario as an example, the reference trajectory includes the first reference question from the interacting party, "I have a fever, can you give me some medication suggestions?", and the corresponding reference response could be, "How many days has your fever lasted?". The interacting party outputs the response corresponding to this reference response as, "The fever has lasted for two days." Subsequently, through a series of reference questions and responses, it can be determined that the patient has had a fever for two days, with a high fever of 38 degrees Celsius, accompanied by symptoms such as nasal congestion and sore throat. Therefore, the final reference response could be, "Your condition may be an upper respiratory tract infection, and it is recommended to use medication A for treatment."

[0097] When the updated agent receives the same exploration question, "I have a fever, can you give me some medication suggestions?", the generated exploration response is, "Your condition may be an upper respiratory tract infection, a viral cold, or a rheumatic cold. It is recommended to use the corresponding medication." The interacting party outputs the response corresponding to this exploration response as, "Can you give me some specific medication suggestions?", thus forming an exploration trajectory.

[0098] This allows us to label the reference response "How many days has your fever lasted?" as positive (guiding the respondent to output more relevant factual information), and the exploratory response "Your condition may be an upper respiratory tract infection, a viral cold, or a rheumatic cold; it is recommended to use appropriate medication" as negative (the response is ambiguous), thus obtaining positive and negative results.

[0099] Subsequently, the target data can be defined as "I have a fever, can you give me some medication suggestions?", and the agent can be trained and updated by combining the positive and negative results obtained from the labeling. This will enable the target agent to learn to prioritize generating guiding responses such as "How many days have you had a fever?" rather than accepting ambiguous information, and ultimately obtain a target agent that can proactively guide users.

[0100] In a shopping scenario, the first reference search term for the interacting party can be "dress". The corresponding reference understanding response can be "The following are recommended dress products for you. If you need more detailed recommendations, you can choose the following style tags: business, commuting, vacation, retro". The responding party outputs the response corresponding to this understanding response as having selected the "retro" style tag. Subsequently, the intelligent agent provides the interacting party with retro-style dress recommendations, enabling the interacting party to select the final dress from the retro-style dresses.

[0101] When the updated agent received the same reference search term "dress", the generated exploration understanding response was "The following are recommended dress products for you". The interaction policy responded to this understanding response by refreshing the recommendation page and eventually exited because no suitable products were found.

[0102] Therefore, we can mark the response "For more detailed recommendations, you can choose the following style tags: business, commuting, vacation, retro" as positive (guiding the interactive party to output more relevant search information) and mark the response "The following are recommended related dress products for you" as negative (not guiding out any relevant information), thus obtaining positive and negative results.

[0103] In one or more embodiments of this specification, when multiple rounds of reference results and multiple rounds of exploration results are obtained, the reference results and exploration results can be labeled to obtain corresponding labeling information, thereby determining positive and negative results based on the labeling information. Specific implementation methods are described below:

[0104] The step of determining positive and negative results based on the annotation information of the multi-round reference results and the multi-round exploration results includes:

[0105] Based on the set of atomic facts of the domain events corresponding to the multi-round reference data and the multi-round exploration data, the annotation information is added to the multi-round reference results and the multi-round exploration results;

[0106] If the annotation information is positive, the reference result and / or exploration result corresponding to the annotation information will be determined as the positive result;

[0107] If the annotation information is negative, the reference result and / or exploration result corresponding to the annotation information will be determined as the negative result.

[0108] The atomic fact set contains all the atomic facts corresponding to the domain event; the annotation information can be understood as the value judgment of the reference result and the exploration result, that is, whether the reference result and the exploration result successfully obtained the atomic fact (positive) or failed to obtain effective information (negative, such as the user answering "I don't know"); a positive result can be understood as a valid result that successfully guides the interaction party to provide a certain atomic fact; while a negative response can be understood as an invalid result that fails to guide the interaction party to provide effective information (such as the interaction party answering "I don't know" or answering irrelevantly).

[0109] Specifically, the reference trajectory and exploration trajectory correspond to events in the same domain. The set of atomic facts corresponding to these events is determined, and it is checked whether the results of each round of interaction in both the reference and exploration trajectories successfully guided the interacting party to provide atomic facts. This means determining whether the respondent's output response to the reference and / or exploration results matches a specific atomic fact in the set of atomic facts. If a match is successful, the reference and / or exploration results that successfully guided the interacting party to provide an atomic fact are marked as positive. If the interacting party's response to the reference and / or exploration results is "don't know" or does not contain any atomic facts, the reference and / or exploration results that failed to guide the interacting party to provide valid information are marked as negative. Thus, positively marked reference and / or exploration results are determined as positive responses, and negatively marked reference and / or exploration results are determined as negative responses, providing a data foundation for subsequently constructing preference data pairs.

[0110] For example, in a legal consultation scenario, consider a traffic accident where the atomic facts include "I damaged the car's bumper" and "I was asked to pay 3,000 yuan in compensation." The user's first question (i.e., the first round of reference question or the first round of exploration question) is "I was in a car accident and was asked to pay compensation. Is this reasonable?" In the reference trajectory, the reference response to the first question could be "What were the specific details of the car accident?" or "How much compensation is the other party asking you to pay?" Based on the reference response, the user outputs the next round of reference questions as "The specific details of the car accident are that I hit the bumper on the side of the road while riding my bike" or "The other party is asking me to pay 3,000 yuan in compensation." Since the reference response to the first question successfully guides the user to output a reference question containing atomic facts, the reference response to the first question can be marked as positive, that is, the reference response to the first question is a positive response.

[0111] Although an updated agent has been trained based on the reference trajectory to obtain an updated agent, the agent can perform diverse exploratory behaviors during actual interactions. Therefore, in the exploration trajectory, the updated agent's exploratory response to the first round of questions could be "If the accident is your fault, you need to compensate." If the user outputs the next round of exploration questions based on the exploration response as "I don't know who is responsible for the accident now, I need you to make a judgment," the user's output exploration question does not contain atomic facts related to the domain event. Therefore, the updated agent's output exploration response to the first round of questions does not guide the user to output effective information, and the exploration response to the first round of questions is marked as negative.

[0112] The agent training method provided in the embodiments of this specification can objectively evaluate reference responses and exploratory responses based on atomic fact sets, and determine whether the reference responses and exploratory responses are positive or negative responses based on the annotation information of the reference responses and exploratory responses, thus providing a data foundation for subsequently constructing high-quality preference data pairs and driving the agent to optimize strategies.

[0113] In one or more embodiments of this specification, to reduce training costs and improve training effectiveness more efficiently, unique and hard-to-obtain atomic facts contained in the reference trajectory and exploration trajectory are identified. In the above embodiments, if the reference result and exploration result can guide the interactive party to output a question containing atomic facts, the corresponding reference result and exploration result are marked as positive. In this embodiment, however, the corresponding reference result and exploration result are marked as positive only when the reference result and exploration result can guide the interactive party to output data containing specific atomic facts. Specific implementation methods are described below:

[0114] The step of adding the annotation information to the multi-round reference results and multi-round exploration results based on the atomic fact set of domain events corresponding to the multi-round reference data and the multi-round exploration data includes:

[0115] From the set of atomic facts, determine the target reference atomic facts contained in the multi-round reference data, and the target exploration atomic facts contained in the multi-round reference data;

[0116] A set of target atomic facts is determined based on the target reference atomic facts and the target exploration atomic facts, and the annotation information is added to the multi-round reference results and the multi-round exploration results based on the set of target atomic facts.

[0117] Among them, target reference atomic facts can be understood as unique atomic facts contained in the reference data output by the interacting party; target exploration atomic facts can be understood as unique atomic facts contained in the exploration data output by the interacting party; target atomic fact set can be understood as a valuable fact set determined by combining target reference atomic facts and target exploration atomic facts. The atomic facts in the target atomic fact set are usually those key facts that were successfully obtained by one trajectory but missed by another trajectory. As a more stringent annotation standard, the target atomic fact set can more finely evaluate the value of each result.

[0118] Specifically, the unique atomic facts (i.e., target reference atomic facts) contained in the multi-round reference data of the reference trajectory are determined from the atomic fact set, and the unique atomic facts (i.e., target exploration atomic facts) contained in the multi-round exploration data of the exploration trajectory are determined from the atomic fact set. The target atomic fact set is obtained by taking the union of the target reference atomic facts and the target exploration atomic facts. This target atomic fact set contains key atomic facts that have unique contributions in the two trajectories and should be of attention. To improve the training effect of the agent, the reference results and exploration results are labeled according to the more targeted target atomic fact set. That is, the reference results and / or exploration results are labeled as positive only when they successfully guide the interaction party to provide atomic facts belonging to the target atomic fact set; otherwise, even if the reference results and / or exploration results successfully guide the interaction party to provide other atomic facts, they will be regarded as low value and labeled as neutral or negative, rather than positive.

[0119] In one or more embodiments of this specification, when adding annotation information to reference results and exploration results based on a target atomic fact set, the annotation is specifically determined based on whether the response corresponding to the reference result or exploration result contains atomic facts from the target atomic fact set. That is, if the response corresponding to the reference result or exploration result contains atomic facts from the target atomic fact set, the corresponding result is labeled as positive; otherwise, it is labeled as negative. Specific implementation methods are described below:

[0120] The step of adding the annotation information to the multi-round reference results and the multi-round exploration results based on the target atomic fact set includes:

[0121] The target reference result is determined from the multiple rounds of reference results, and the target exploration result is determined from the multiple rounds of exploration results;

[0122] From the multi-round reference data, determine the response reference data corresponding to the target reference result. If the response atomic facts contained in the response reference data belong to the target atomic fact set, add positive annotation information to the target reference result; otherwise, add negative annotation information to the target reference result.

[0123] From the multi-round exploration data, determine the response exploration data corresponding to the target exploration result. If the response atomic facts contained in the response exploration data belong to the target atomic fact set, add positive annotation information to the target exploration result; otherwise, add negative annotation information to the target exploration result.

[0124] Continue executing the steps of determining the target reference result from the multi-round reference results and determining the target exploration result from the multi-round exploration results until the annotation information is added to the multi-round reference results and the multi-round exploration results.

[0125] Here, the target reference result can be understood as any one of the reference results selected from multiple rounds to be labeled; the target exploration result can be understood as any one of the exploration results selected from multiple rounds to be labeled; the response reference data can be understood as the response content output by the interacting party in response to the target reference result in the reference trajectory, and this response content will be used as the reference data for the next round; the response exploration data can be understood as the response content output by the interacting party in response to the target exploration result in the exploration trajectory, and this response content will be used as the exploration data for the next round; and the response atomic facts can be understood as the atomic facts contained in the response reference data or response exploration data.

[0126] Specifically, taking the dialogue scenario as an example, the multi-turn responses (including multi-turn reference responses and multi-turn exploration responses) in the reference trajectory and exploration trajectory are labeled to obtain the labeling information corresponding to the multi-turn responses. In specific implementation, the multi-turn responses in the two trajectories are traversed. When a target response (target reference response or target exploration response) is selected, the response content of the interaction policy to the target response (responding to the reference question or responding to the exploration question) is determined, and the response atomic facts contained in the response content are clarified.

[0127] The response atomic facts are compared with the high-value target atomic facts set obtained above. If the response atomic facts belong to the target atomic facts set, it means that the target response has successfully guided the interacting party to output key atomic facts, and the target response is marked as positive, that is, positive annotation information is added to the target response. If the response atomic facts do not belong to the target atomic facts set, it means that the target response has failed to successfully guide the interacting party to output key fact information, and the target response is marked as negative, that is, negative annotation information is added to the target response. This process is repeated until annotation information is added to the multi-round reference results and multi-round exploration results.

[0128] The agent training method provided in the embodiments of this specification accurately evaluates multi-round responses. By employing step-level action effect attribution, it performs fine-grained identification and labeling of clarification actions in the exploration trajectory and reference trajectory, breaking through the coarse-grained stage that relies on the overall trajectory to determine superiority or inferiority. Furthermore, by labeling based on the target atomic fact set, it ensures that subsequent preference learning focuses on optimizing questioning strategies that can successfully elicit high-value information, thereby guiding the agent to learn to prioritize extracting the most critical information in dialogue, greatly improving the targeting and effectiveness of strategy optimization.

[0129] In specific implementation, the target reference atomic facts and target exploration atomic facts in the target atomic fact set are the key atomic facts. Specifically, based on the reference data output by the interacting party, the atomic facts contained in the reference data are matched from the complete atomic fact set to obtain the reference atomic fact set. Similarly, based on the exploration data output by the interacting party, the atomic facts contained in the exploration data are matched from the complete atomic fact set to form the exploration atomic fact set. Thus, by calculating the difference, the unique atomic facts (i.e., key atomic facts) collected by the two trajectories are determined. The specific implementation method is as follows:

[0130] The step of determining the target reference atomic facts contained in the multi-round reference data and the target exploration atomic facts contained in the multi-round reference data from the atomic fact set includes:

[0131] Based on the multi-round reference data and the atomic fact set, determine the reference atomic fact set corresponding to the multi-round reference data;

[0132] Based on the multi-round exploration data and the atomic fact set, the exploration atomic fact set corresponding to the multi-round exploration data is determined, wherein the reference atomic fact set and the exploration atomic fact set are subsets of the atomic fact set;

[0133] Based on the difference between the reference atomic fact set and the exploration atomic fact set, the target reference atomic fact is determined from the reference atomic fact set, and the target exploration atomic fact is determined from the exploration atomic fact set, wherein the target reference atomic fact belongs to the reference atomic fact set but not to the exploration atomic fact set, and the target exploration atomic fact belongs to the exploration atomic fact set but not to the reference atomic fact set.

[0134] The reference atomic fact set can be understood as the set of atomic facts provided by the reference data output by the interacting party in multiple rounds of interaction on the reference trajectory; the exploration atomic fact set can be understood as the set of atomic facts provided by the exploration data output by the interacting party in multiple rounds of interaction on the exploration trajectory; the target reference atomic fact can be understood as the atomic fact with special value (which the exploration trajectory failed to obtain) selected from the reference atomic fact set by calculating the difference between the reference atomic fact set and the exploration atomic fact set; the target exploration atomic fact can be understood as the atomic fact with special value (which the reference trajectory failed to obtain) selected from the exploration atomic fact set by calculating the difference between the reference atomic fact set and the exploration atomic fact set.

[0135] Based on the reference questions output by the interacting parties in the reference trajectory, the atomic facts corresponding to the reference questions are determined from the complete set of atomic facts to obtain the reference atomic fact set. Based on the exploration questions output by the interacting parties in the exploration trajectory, the atomic facts corresponding to the exploration questions are determined from the complete set of atomic facts to obtain the exploration atomic fact set. By comparing and analyzing these two sets (using set difference operation), valuable atomic facts are selected.

[0136] In practical applications, atomic facts that are included in the reference atomic fact set but not in the exploration atomic fact set are identified as target reference atomic facts (i.e., unique atomic facts included in the reference atomic fact set), which represents the uniqueness of the expert strategy; correspondingly, atomic facts that are included in the exploration atomic fact set but not in the reference atomic fact set are identified as target exploration atomic facts (i.e., unique atomic facts included in the exploration atomic fact set), which represents the potential new strategies discovered in the agent's exploration.

[0137] For example, for the first round of problem s0, the reference trajectory is (s0, a1) exp s1 exp a2 exp s2 exp The exploration trajectory is (s0, a1), ...), pro s1 pro a2 pro s2 pro ,…), where s exp The reference question output by the interacting party, a exp For reference, s pro The exploration question output by the interacting party, a pro For exploring responses. Each s may contain an atomic fact or "unknown" (representing invalid information that does not contain an atomic fact).

[0138] Given that the set of atomic facts for a domain event includes {f1, f2, f3, f4, f5} (where f represents an atomic fact), in the reference trajectory a1 exp a2 exp a3 exp a4 exp Corresponding to s1 exp s2 exp s3 exp s4 exp The information contained are (f1, f2, unknown, f3), and the exploration trajectory contains a1. pro a2 pro a3 pro a4 pro Corresponding to s1 pro s2 pro s3 pro s4 pro The information contained are (f1, f4, unknown, f3).

[0139] Therefore, based on the multi-round reference problem and the set of atomic facts, the reference atomic fact set can be determined as {f1, f2, f3}, and the exploration atomic fact set can be determined as {f1, f4, f3}. By comparing the reference atomic fact set and the exploration atomic fact set and calculating the difference, the target reference atomic fact can be determined as f2, and the exploration reference atomic fact as f4. Since both sets contain f1 and f3, it indicates that a1 in the reference trajectory... exp a4 exp and in the exploration trajectory a1 pro a4 pro The effect is the same.

[0140] Because of a2 in the reference trajectory exp This reference response successfully guided the interacting party to output the target reference atom fact f2, thus a2 exp Marked as positive, in the exploration trajectory a2 pro This exploratory response successfully guided the interacting party to output the target exploratory atomic fact f4, thus a2 pro It is also marked as positive. In the reference trajectory, a3... exp and in the exploration trajectory a3 pro Neither approach successfully guided the interacting party to output valid atomic facts, therefore a3 was... exp and a3 pro Marked as negative.

[0141] The agent training method provided in the embodiments of this specification automatically filters out the key atomic facts (i.e., atomic facts that one trajectory has but another trajectory does not) that best distinguish the superiority or inferiority of a strategy by calculating the set difference. This allows the subsequent agent optimization to focus on learning those challenging clarifying actions that reflect the policy gap, thereby more efficiently improving the agent's ability to actively explore and acquire key information, avoiding redundant learning, and significantly improving training efficiency and the final performance of the agent.

[0142] Step 208: Train the updated agent based on the target data, the positive results, and the negative results to obtain the target agent, wherein the target data is reference data or exploration data.

[0143] Specifically, guided by the target question, preference data pairs are constructed by combining positive and negative responses to further optimize the agent's strategy, ultimately resulting in a high-performance target agent.

[0144] In one or more embodiments of this specification, preference data pairs are constructed by obtaining the target question, positive response, and negative response. The agent is then trained and updated based on these preference data pairs containing preference information, enabling the obtained target agent to absorb clarification actions that perform well in the reference trajectory and exploration trajectory, thereby enhancing its ability to actively identify and fill information gaps. Specific implementation methods are described below:

[0145] The step of training the updated agent based on the target data, the positive results, and the negative results to obtain the target agent includes:

[0146] Based on the target data corresponding to the positive result and the off-track negative result determined from the negative result, a first data pair corresponding to the positive result is constructed, wherein the positive result and the off-track negative result have different sources;

[0147] Based on the target data corresponding to the negative result and the off-track positive result determined from the positive result, a second data pair corresponding to the negative result is constructed;

[0148] The update agent is trained based on the first data pair corresponding to the positive result and the second data pair corresponding to the negative result to obtain the target agent.

[0149] Specifically, the first data pair corresponding to a positive result can be understood as the positive sample training data for training and updating the agent, and the second data pair corresponding to a negative result can be understood as the negative sample training data for training and updating the agent.

[0150] In specific implementation, a target positive response is determined from the positive responses, and the target question corresponding to the target positive response is determined. A first data pair is constructed based on the target question corresponding to the target positive response, the target positive response, and the off-track negative response. The target positive response is any one of the positive responses, and the target positive response and the off-track negative response originate from different trajectories.

[0151] Continue executing the step of determining the target positive response from the positive responses until the first data pair corresponding to each positive response is obtained;

[0152] Determine a target negative response from the negative responses, determine the target question corresponding to the target negative response, and construct a second data pair based on the target question corresponding to the target negative response, the off-track positive response, and the target negative response. The target negative response is any one of the negative responses, and the target negative response and the off-track positive response originate from different trajectories.

[0153] Continue executing the step of determining the target negative response from the negative responses until the second data pair corresponding to each negative response is obtained.

[0154] Here, a target positive response can be understood as any positive response selected from those marked as positive; a target negative response can be understood as any negative response selected from those marked as negative; a target question can be understood as a reference or exploratory question raised by the interacting party that triggers a target positive or negative response; the first data pair and the second data pair are both data pairs composed of a triple of target question, positive response, and negative response. The target question in the first data pair is the question corresponding to the positive response, and the target question in the second data pair is the question corresponding to the negative response.

[0155] Specifically, the process iterates through positive responses to determine a target positive response. The target question corresponding to this positive response is then identified from the reference or exploration trajectory. This target positive response is the response to the target question. Since positive and negative responses are determined by labeling the reference and exploration responses, the trajectories corresponding to the positive and negative responses can be determined. It should be noted that when constructing the first data pair and determining the negative response to the target positive response pair, it is necessary to ensure that the target positive response and the selected negative response originate from different trajectories. That is, a non-target negative response from a different trajectory is selected from the negative responses. For example, if the target positive response originates from the reference trajectory, then a negative response from the exploration trajectory is selected as a non-target negative response. Thus, the target question, the target positive response, and the non-target negative response are combined to form the first data pair corresponding to the target positive response. The implicit preference information of this first data pair is that the agent should tend to generate this type of target positive response when facing a target question.

[0156] Following the above embodiments, when a2 exp and a2 pro If the label is positive, then determine a2. exp and a2 pro As a positive response, a2 exp As a positive response to the target, from a2 exp The target problem is determined as reference problem s1 from the corresponding reference trajectory. exp That is, the target positive response a2 exp For reference problem s1 exp The generated response, in the negative response, contains a3 exp and a3 pro In this case, choose the exploration trajectory a3 pro As a negative response to the off-track, the first data pair (s1) is constructed. exp a2 exp a3 pro ), having obtained the target agent through training using this first data, the target agent, when faced with s1 exp In such cases, a2 should be generated first. exp Similar positive responses, while avoiding the generation of a3. exp or a3 exp The negative response. Similarly, when a2 pro As a positive response to the target, construct the first data pair (s1) pro a2 pro a3 exp ).

[0157] The process involves iterating through negative responses to identify a target negative response. The target question corresponding to this negative response is then determined from either the reference trajectory or the exploration trajectory. This target negative response is the response to the target question. Similar to constructing the first data pair, when constructing the second data pair and determining the positive response paired with the target negative response, it is ensured that the target negative response and the selected positive response originate from different trajectories. Specifically, a different-track positive response is selected from the positive responses. For example, if the target negative response originates from the exploration trajectory, a positive response from the reference trajectory is selected as the different-track positive response. Thus, based on the target question corresponding to the target negative response, the different-track positive response, and the target negative response, a second data pair corresponding to the target negative response is formed. The implicit preference information of this second data pair is that the agent should avoid generating such a target negative response when facing a target question.

[0158] Following the above embodiments, when a3 exp and a3 pro When the label is negative, determine a3. exp and a3 pro As a negative response, a3 exp As a negative response to the target, from a3 exp The target problem is determined as reference problem s2 from the corresponding reference trajectory. exp That is, the target negative response a3 exp For reference problem s2 exp The generated response, in the positive response containing a2 exp and a2 pro In this case, choose the exploration trajectory a2 pro As a positive response to the off-track, a second data pair (s2) is constructed. exp a2 pro a3 exp Thus, when the target agent is obtained through training using this first data, the target agent, when faced with s2 exp In such cases, a2 should be generated first. pro Similar positive responses, while avoiding the generation of a3. exp The negative response. Similarly, when a3 pro In the case of a negative response to the target, a second data pair (s2) is constructed. pro a2 exp a3 pro ).

[0159] That is, by using the obtained first and second data pairs, the agent is trained through a contrastive learning algorithm (such as DPO) to continuously adjust its policy toward generating positive responses and avoiding negative responses, thereby obtaining the final target agent.

[0160] The agent training method provided in the embodiments of this specification constructs data pairs centered on positive responses and negative responses. During the training process, it not only strengthens the agent's imitation and learning of efficient questioning strategies, but also explicitly suppresses undesirable behaviors. This bidirectional comparative optimization mechanism can guide the agent to master the core ability of proactive clarification more quickly and stably, enabling it to autonomously generate high-value responses and significantly improve the efficiency and effectiveness of dialogue.

[0161] The agent training method provided in the embodiments of this specification utilizes domain event patterns to abstract and normalize the factual information provided by the interacting parties, thereby realizing structured knowledge-driven dialogue planning. Furthermore, through a step preference attribution optimization method, it can automatically identify and strengthen key clarification actions. By integrating expert demonstrations and agent autonomous exploration, it forms a continuously optimized and highly generalizable agent training framework. Specifically, this enables the agent not only to fully grasp the existing action chains of experts but also to flexibly absorb newly discovered effective questioning methods during exploration. This allows it to demonstrate clarification and innovation capabilities that surpass expert demonstrations in unknown or new scenarios, greatly improving its adaptability and generalization to complex, dynamic, and multi-round expert consultation tasks. Based on the obtained target agent, it can provide high-quality and efficient services in various scenarios.

[0162] See Figure 3 , Figure 3 A flowchart of a data processing method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0163] Step 302: Input the initial data into the target agent to obtain the initial result output by the target agent, wherein the target agent is obtained according to the above-described agent training method.

[0164] The target agent can be understood as an agent with efficient clarification capabilities obtained after preference optimization; the initial data can be understood as the question raised by the interacting party at the beginning, which is usually vague or lacks information; the initial result can be understood as the response generated by the target agent in response to the initial data.

[0165] Specifically, the training process for the target agent is explained in detail: By using reference trajectories containing multiple rounds of reference questions and corresponding reference responses, the updated agent can better understand contextual semantics and reasoning structures, improving its grasp of domain knowledge, inductive ability, and multi-stage strategy planning capabilities. The updated agent interacts with the interacting party, generating diverse exploration trajectories through autonomous exploration. This results in exploration trajectories containing multiple rounds of exploration questions and corresponding exploration responses. The updated agent can not only imitate expert behavior but also produce novel dialogue paths. Attribution evaluation is performed on each clarification action in the reference and exploration trajectories, accurately identifying unique and effective clarification actions in each trajectory. Reference and exploration responses are labeled, and positive and negative responses are obtained, overcoming the coarse-grained approach that relies on the overall trajectory for ranking. In fact, both reference and exploration responses in the reference and exploration trajectories can potentially be guiding responses. When training the agent to enhance its clarification capabilities, if a response guides the interacting party to output a reference question containing atomic facts (i.e., the response is a guiding response), it is labeled as positive, thus identifying it as a positive response. The agent is trained and updated based on the target question, positive response, and negative response to obtain the target agent. The quality feedback of the action (response) is directly applied to the agent's policy update, realizing the automatic inheritance and generalization of positive actions, and significantly accelerating the agent's absorption and iteration of high-quality clarification strategies.

[0166] When using the target intelligent agent to generate initial results corresponding to the initial data, it can generate positive initial results that can guide users to output more effective information, or generate accurate initial results that can answer the initial data.

[0167] Step 304: In response to the response data corresponding to the initial result, determine the response data as the initial data, and continue to execute the step of inputting the initial data into the target agent until the stopping condition is met.

[0168] The response data can be understood as the new data or feedback information given by the user in response to the previous round of responses from the intelligent agent; the stopping condition can be understood as the pre-set interaction termination condition, which includes, but is not limited to, the target intelligent agent giving a final answer based on sufficient information collected, the interaction rounds reaching the upper limit, or the user actively ending the interaction process, etc., without further limitation.

[0169] Specifically, after the target agent provides an initial result, the user responds by outputting new input data (i.e., response data), updating the "initial data" for the next round of interaction. Then, the initial data is input into the same target agent again to obtain a new initial result. This process is repeated until a preset stopping condition is met, thus completing a full and continuous multi-round interaction process.

[0170] In fact, the target agent can dynamically process the ever-changing dialogue context and adjust its strategy in real time based on the user's latest feedback, thereby providing more in-depth guidance or giving more accurate answers.

[0171] The data processing method provided in the embodiments of this specification enables the target intelligent agent to not only fully grasp the existing action links of experts, but also flexibly absorb newly discovered effective questioning methods during exploration. Thus, it can demonstrate a clarifying and innovative ability that surpasses expert demonstrations in unknown or new scenarios, greatly improving the adaptability and generalization of complex, dynamic, and multi-round data processing tasks.

[0172] See Figure 4 , Figure 4 A flowchart of a question-and-answer method provided in one embodiment of this specification is shown, which specifically includes the following steps.

[0173] Step 402: Input the initial question into the target agent and obtain the initial response output by the target agent, wherein the target agent is obtained according to the above-described agent training method.

[0174] The initial question can be understood as the question raised by the user at the very beginning, which is usually vague or lacks information; the initial response can be understood as the response generated by the target agent in response to the initial question, which is usually a high-value clarifying question or guiding action.

[0175] Specifically, a user poses an initial, vague question, which the target agent interprets and generates a high-quality initial response. This target agent is optimized using dialogue data generated from expert demonstrations (reference trajectories) and its own exploration (exploration trajectories) through contrastive learning. The training data for the target agent includes the target question, positive responses, and negative responses. The target question originates from the reference trajectories and user questions within those trajectories. Positive and negative responses are determined by assigning value labels (whether they can guide the user to provide key atomic facts) to the responses to the user questions. Thus, preference data pairs are constructed based on the target question, positive responses, and negative responses, enabling the target agent to be trained using contrastive learning.

[0176] It should be noted that when the reference response or exploratory response is a guiding response, the guiding question will be marked as positive. When the training data includes guiding responses, the guiding response enables the target agent to learn not to rush to answer the user's question directly, but to prioritize guiding the user to supplement key information through questioning, thereby laying the foundation for accurate answers later.

[0177] In other words, when the target intelligent agent receives an initial question that is vague and lacks information, its initial response is usually a guiding response, which guides the user to provide more information related to the inquiry.

[0178] Step 404: In response to the response question corresponding to the initial response, determine the response question as the initial question, and continue to execute the step of inputting the initial question into the target agent until the stopping condition is met.

[0179] The response question can be understood as a new question or feedback information given by the user in response to the agent's previous reply; the stopping condition can be understood as a pre-set dialogue termination condition, which includes, but is not limited to, the target agent giving a final answer based on sufficient information collected, the dialogue rounds reaching the limit, or the user actively ending the dialogue, etc., without further limitation.

[0180] Specifically, after the target agent provides an initial response, the user responds by outputting a new input question (i.e., a response question), which is then updated as the initial question for the next round of dialogue. The initial question is then input into the same target agent again to obtain a new response. This process is repeated until a preset stopping condition is met, thus completing a full, continuous, multi-round consultation process.

[0181] In fact, the target agent can dynamically process the ever-changing dialogue context and adjust its strategy in real time based on the user's latest feedback, thereby providing more in-depth guidance or giving more accurate answers.

[0182] The question-and-answer method provided in the embodiments of this specification, when using a target intelligent agent to provide question-and-answer services, compared with the traditional intelligent agent's passive response mode, which often hastily answers when information is incomplete, the target intelligent agent adopts a strategy of actively guiding and clarifying questions, thereby reducing invalid dialogue rounds caused by ambiguous information, significantly improving the efficiency of problem solving and user satisfaction, and achieving professional-level efficient consultation interaction.

[0183] See Figure 5 , Figure 5 The flowchart of a question-answering method applied to a question-answering platform according to an embodiment of this specification is shown, which specifically includes the following steps.

[0184] Specifically, the question-and-answer platform deploys target intelligent agents.

[0185] Step 502: In response to the query request sent by the client, obtain the initial question carried in the query request.

[0186] Step 504: Input the initial question into the target agent, obtain the initial response output by the target agent, and return the initial response to the client, wherein the target agent is obtained according to the above agent training method.

[0187] Step 506: In response to the follow-up question sent by the client corresponding to the initial response, obtain the response question corresponding to the initial response, determine the response question as the initial question, and continue to execute the step of inputting the initial question into the target agent until the stopping condition is met.

[0188] When a target agent is deployed on a question-and-answer platform, the platform provides question-and-answer services. When a user accesses the platform through a client and sends an initial question, the platform responds to the client's inquiry request, obtains the initial question, and uses the target agent to generate an initial response. After returning the initial response to the client and displaying it to the user, the user can send follow-up questions about the initial response. These follow-up questions carry the corresponding response questions, which the target agent processes to generate a response. Further rounds of dialogue can then be conducted until the target agent provides a final answer based on sufficient information, the maximum number of dialogue rounds is reached, or the user actively ends the dialogue.

[0189] For specific implementation methods, please refer to the above embodiments, which will not be repeated here.

[0190] See Figure 6 , Figure 6 The flowchart of a smart agent training method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0191] The agent training method provided in the embodiments of this specification combines knowledge modeling of domain event patterns and proposes a step preference attribution optimization algorithm to construct a training framework for proactive consultation agents for consultation scenarios such as law and medicine.

[0192] Step 602: Behavioral cloning guided by domain event patterns.

[0193] Specifically, in consultation scenarios, when users provide atomic facts (such as specific event descriptions or background information) in their questions, the domain event patterns summarized by domain experts are used to normalize and map the atomic facts. Specifically, through predefined schema tags (such as <amount>, <injury time>, etc.), the originally complex and trivial user statements are abstracted into higher-order requirement attributes, thereby establishing a structured knowledge base.

[0194] In practical implementation, for the atomic facts k∈K (where K represents the set of atomic facts of domain events) in the user's question (i.e., the reference question), an event pattern label set L defined by the domain expert (i.e., the label set corresponding to the domain events in the above embodiment) is introduced, and a mapping function is used. Each atomic fact is mapped to a higher-level scheme label (i.e., the fact label in the above embodiment).

[0195] Based on the obtained mapping results, the system automatically invokes the reasoning chain of the large model according to the domain event pattern, specifically utilizing functions. The system systematically generates clarifying questions (also known as clarifying actions, i.e. expert actions, which can be understood as reference responses in the above embodiments). Clarifying questions refer to what kind of questions experts can ask to elicit the corresponding atomic facts. By generating clarifying questions, the system can ensure full coverage of facts in key areas and effectively reflect the expert's decision-making logic, thereby enabling the reference trajectory to possess the systematic nature of expert knowledge.

[0196] Each round of clarification questions and user responses forms a complete expert trajectory. , ,in, For the user's initial question, the last expert action in the trajectory. This indicates the final recommendation generated; the expert trajectory is the reference trajectory in the above embodiment. The entire expert trajectory uses domain schema tags as clues to realize expert consultation dialogue data from multiple rounds of clarifying questions to giving the final professional advice. By serializing the context state, expert action 'a', and user response 's' in the dialogue, high-quality expert data is formed.

[0197] During behavior cloning training, the expert agent learns standard dialogue actions from expert trajectories, thereby better understanding contextual semantics and reasoning structures, and improving its grasp of domain knowledge, inductive ability, and multi-stage policy planning ability.

[0198] The goal of behavioral cloning is Among them, the agent is optimized using the SFT (Supervised Fine-Tuning) loss function, and main indicates that it is the core component of the total loss function. This represents the policy of the agent to be trained. Indicates from expert dataset Expert trajectory of mid-sample Expectations Indicates the first The length of the trajectory (time steps). For time step index, Indicates the first In the trajectory, the first Expert actions at each time step; Indicates the first In the trajectory, the first The state (user response) up to each time step, including until... All dialogue history and context up to step; Represents a given state Below, agent strategy Select Expert Actions The probability of.

[0199] In fact, the initial intelligent agent, the updated intelligent agent, and the target intelligent agent in the above embodiments have the same structure. These three are different stages of the intelligent agent and have different capabilities. When the expert intelligent agent has learned the expert trajectory, the expert intelligent agent at this time is the updated intelligent agent in the above embodiments.

[0200] Step 604: Step preference attribution optimization enhances proactive exploration capabilities.

[0201] After the agent possesses the basic capabilities of expert linking, it is allowed to autonomously explore with the environment or users (i.e., the interacting parties), generating diverse exploration trajectories. This is to capture more potential clarification strategies and variations in actual interactions.

[0202] For each expert trajectory and exploration trajectory, based on the step preference attribution optimization mechanism (SPAO), the information obtained by each clarification action in different trajectories is automatically compared to determine which actions can independently uncover the missed atomic facts.

[0203] Specifically, for each trajectory τ, the key information (i.e., atomic facts) it acquires is determined. ,in, This represents the action performed at time step t. Indicates time step state, Functions represent states Execute The feedback or response information obtained, such as actions in a dialogue scenario. It might be a guiding question, and This indicates that the user has responded effectively to the guided question. If no valid response is elicited, an empty set is returned. . This represents the union operation, which collects data from all time steps. The result, and remove the empty set (i.e. In fact, this step allows us to obtain the set of reference atomic facts corresponding to the reference trajectory and the set of exploration atomic facts corresponding to the exploration trajectory in the above embodiments.

[0204] By comparing the expert trajectory and the exploration trajectory, the unique atomic facts of each trajectory are obtained (i.e., the target reference atomic facts and the target exploration atomic facts in the above embodiments). This represents the set of information (i.e., target reference atomic facts) that the expert trajectory successfully acquired but the exploration trajectory missed. This represents the set of information (i.e., target exploration atomic facts) that were successfully acquired by the exploration trajectory but not included in the expert trajectory.

[0205] Based on the fine-grained attribution of each action step described above, preference samples of "preferred action - inferior action" are automatically labeled and constructed in pairs, enabling precise learning and incentives for high-value clarification questions. Specifically, for each action... in, This represents a positive feedback set (i.e., the target atomic fact set in the above embodiments), through the above... as well as The union of the sets is used to obtain the results. Pre represents the accept label (i.e., labeled as positive) and Rej represents the reject label (i.e., labeled as negative).

[0206] That is, when an action In state During execution, the Elicits function determines the feedback corresponding to the action. If the corresponding feedback belongs to the positive feedback set, the action is executed. Marked as positive, and determined as the preferred action a w (That is, the action successfully extracted key information); if the feedback for the action is empty and no valid information is obtained, then the action is terminated. Marked as negative, it is determined to be the inferior choice action a. l (That is, the action failed to guide the user to output a valuable response).

[0207] Combining the preference samples of "optimal action - inferior action" and the state s to construct (s, a w a lPreference data pairs; and based on comparative preference optimization paradigms such as DPO, the updating agent is optimized in terms of strategy, so that it can continuously absorb the clarification actions that perform well in the expert trajectory and exploration trajectory, gradually enhance its ability to actively identify and fill information gaps, and obtain a high-performance target agent.

[0208] in, Represents the DPO loss function. The reference strategy refers to the strategy for updating the agent obtained through expert trajectory training in the above embodiments. Indicates from preference dataset The expectation of sampling in the middle, For the triple (s, a) w a l A dataset consisting of ) Used to measure agent policies Compared to the reference strategy For a w The degree of preference; Used to measure agent policies Compared to the reference strategy For a l The degree of preference. The purpose of this loss function is to maximize the agent's policy. Compared to the reference strategy Prefer to select action a w Instead of the inferior choice action a l The probability of that.

[0209] This agent training method uses an automated exploration-optimization closed loop, iterates repeatedly, continuously discovers high-quality clarification strategies and internalizes them into the agent's long-term capabilities, achieving self-evolution and generalization for complex consulting scenarios.

[0210] Specifically, by introducing domain event patterns, the system automatically normalizes the user's original facts into high-level schema labels and systematically generates high-coverage clarification actions based on these labels. This ensures that, with the expert trajectory organized around the schema, each round of question-and-answer can be grounded in the knowledge context, effectively improving the agent's domain understanding, context awareness, and dynamic decision-making capabilities. It can also recreate the real reasoning chain of domain experts, significantly improving the ability to parse fuzzy user input and enabling multi-step information completion and in-depth clarification planning.

[0211] With the acquisition of updated agents through training, the updated agents are allowed to autonomously explore and generate diverse exploration trajectories with the environment. This means that they can not only imitate expert behavior but also produce novel dialogue paths. Based on the SPAO step preference attribution optimization mechanism, the system automatically evaluates each clarification action in the expert trajectory and exploration trajectory, accurately discovering unique and effective clarification actions in the exploration path, thus breaking through the coarse-grained link that relies on the overall trajectory to determine the merits.

[0212] By using DPO comparative optimization, the feedback on the merits and demerits of key steps is directly applied to policy updates, enabling the automatic inheritance and generalization of optimal actions. This significantly accelerates the absorption and iteration of high-quality clarification strategies by the agent, allowing the trained target agent to not only fully grasp the existing action chains of experts but also flexibly absorb newly discovered effective questioning methods during exploration. As a result, it demonstrates clarification and innovation capabilities that surpass expert demonstrations in unknown or new scenarios, greatly improving the adaptability and generalization of complex, dynamic, and multi-round expert consultation tasks.

[0213] The agent training method provided in the embodiments of this specification introduces a domain event pattern-guided behavior cloning method. This method abstracts and normalizes user-provided atomic facts using domain event schema tags, driving behavior cloning. This addresses the difficulty of lacking training data for clarifying questions in consultation scenarios, significantly improving the agent's understanding and clarification planning capabilities regarding complex domain knowledge chains. Furthermore, it employs a step-preference attribution optimization mechanism to perform fine-grained superiority and inferiority identification and reinforcement optimization of clarification actions in exploration and expert trajectories, achieving automatic discovery, inheritance, and generalization of key clarification strategies. Finally, it combines expert demonstration with the agent's autonomous exploration, forming an iterative optimization process for optimal clarification strategies, significantly improving the model's proactive clarification capabilities and generalization performance.

[0214] In practical applications, structured domain knowledge is combined with intelligent agents capable of autonomous exploration and clarification to enhance the agents' multi-round interaction and decision-making capabilities in complex expert consultation tasks. By combining domain event pattern knowledge modeling and step preference attribution optimization algorithms, a training framework for proactive consultation agents in legal, medical, and other consultation scenarios is constructed. This greatly improves the clarification capabilities and decision-making quality of the target intelligent agents, providing a new paradigm for the digital transformation of traditional knowledge service links. It enhances the automated clarification and multi-round interaction capabilities of target intelligent agents in legal, medical, and other expert consultation scenarios, reduces reliance on human experts, and improves consultation efficiency and quality.

[0215] Corresponding to the above method embodiments, this specification also provides embodiments of an intelligent agent training device. Figure 7 A schematic diagram of the structure of an agent training device according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0216] The first training module 702 is configured to train the initial agent using multiple rounds of reference data and the corresponding reference results, and obtain the updated agent.

[0217] The exploration module 704 is configured to obtain multiple rounds of exploration data and corresponding exploration results based on the updated agent, wherein the first round of exploration data is the same as the first round of reference data.

[0218] The annotation module 706 is configured to determine positive and negative results based on the annotation information of the multi-round reference results and the multi-round exploration results, wherein the guidance results included in the reference results and / or exploration results are positive results, and the guidance results are used to guide the interactive party to output data containing atomic facts;

[0219] The second training module 708 is configured to train the updated agent based on the target data, the positive results, and the negative results to obtain the target agent, wherein the target data is reference data or exploration data.

[0220] The apparatus further includes: a data acquisition module configured to determine reference data and fact tags corresponding to the reference data, wherein the reference data is the reference data of the first round in the multi-round reference data, the multi-round reference data corresponds to a domain event containing one or more atomic facts, and the fact tags are mapping tags corresponding to the atomic facts; generate a reference result corresponding to the reference data based on the reference data and the fact tags, wherein the reference result is the guidance result or the solution result; update the reference data to the response reference data in response to the response reference data corresponding to the reference result, and perform round iterations by executing the step of determining the reference data and the fact tags corresponding to the reference data to obtain the multi-round reference data and the corresponding reference result.

[0221] Optionally, the data acquisition module is further configured to: determine the reference data and the domain event corresponding to the reference data; and determine the fact tag corresponding to the reference data from the tag set corresponding to the domain event.

[0222] Optionally, the exploration module 704 is further configured to: input exploration data into the updating agent to obtain the exploration result output by the updating agent, wherein the exploration data is the exploration data of the first round in the multi-round exploration data, and the multi-round exploration data corresponds to a domain event containing one or more atomic facts; in response to the response exploration data output by the interacting party corresponding to the exploration result, update the exploration data to the response exploration data, and perform round iterations by executing the step of inputting exploration data into the updating agent to obtain the multi-round exploration data and the corresponding exploration result.

[0223] Optionally, the annotation module 706 is further configured to: add the annotation information to the multi-round reference results and the multi-round exploration results based on the atomic fact set of the domain events corresponding to the multi-round reference data and the multi-round exploration data; if the annotation information is positive, determine the reference result and / or exploration result corresponding to the annotation information as the positive result; if the annotation information is negative, determine the reference result and / or exploration result corresponding to the annotation information as the negative result.

[0224] Optionally, the annotation module 706 is further configured to: determine the target reference atomic facts contained in the multi-round reference data and the target exploration atomic facts contained in the multi-round reference data from the atomic fact set; determine the target atomic fact set based on the target reference atomic facts and the target exploration atomic facts; and add the annotation information to the multi-round reference results and the multi-round exploration results based on the target atomic fact set.

[0225] Optionally, the annotation module 706 is further configured to: determine a set of reference atomic facts corresponding to the multi-round reference data based on the multi-round reference data and the set of atomic facts; determine a set of exploration atomic facts corresponding to the multi-round exploration data based on the multi-round exploration data and the set of atomic facts, wherein the set of reference atomic facts and the set of exploration atomic facts are subsets of the set of atomic facts; determine the target reference atomic fact from the set of reference atomic facts and the target exploration atomic fact from the set of exploration atomic facts based on the difference between the set of reference atomic facts and the set of exploration atomic facts, wherein the target reference atomic fact belongs to the set of reference atomic facts but not to the set of exploration atomic facts, and the target exploration atomic fact belongs to the set of exploration atomic facts but not to the set of reference atomic facts.

[0226] Optionally, the annotation module 706 is further configured to: determine a target reference result from the multi-round reference results and determine a target exploration result from the multi-round exploration results; determine response reference data corresponding to the target reference result from the multi-round reference data, and add positive annotation information to the target reference result if the response atomic facts contained in the response reference data belong to the target atomic fact set, otherwise add negative annotation information to the target reference result; determine response exploration data corresponding to the target exploration result from the multi-round exploration data, and add positive annotation information to the target exploration result if the response atomic facts contained in the response exploration data belong to the target atomic fact set, otherwise add negative annotation information to the target exploration result; continue to execute the steps of determining the target reference result from the multi-round reference results and determining the target exploration result from the multi-round exploration results until the annotation information is added to the multi-round reference results and the multi-round exploration results.

[0227] Optionally, the second training module 708 is further configured to: construct a first data pair corresponding to the positive result based on the target data corresponding to the positive result and the off-track negative result determined from the negative result, wherein the positive result and the off-track negative result have different sources; construct a second data pair corresponding to the negative result based on the target data corresponding to the negative result and the off-track positive result determined from the positive result; and train the updating agent according to the first data pair corresponding to the positive result and the second data pair corresponding to the negative result to obtain the target agent.

[0228] The above is an illustrative scheme of an agent training device according to this embodiment. It should be noted that the technical solution of this agent training device and the technical solution of the agent training method described above belong to the same concept. For details not described in detail in the technical solution of the agent training device, please refer to the description of the technical solution of the agent training method described above.

[0229] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 8 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 8 As shown, the device includes:

[0230] The result acquisition module 802 is configured to input initial data into the target agent and obtain the initial result output by the target agent, wherein the target agent is obtained according to the above-described agent training method;

[0231] The data update module 804 is configured to respond to the response data corresponding to the initial result, determine the response data as the initial data, and continue to execute the step of inputting the initial data into the target agent until the stopping condition is met.

[0232] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0233] Figure 9 A structural block diagram of a computing device 900 provided according to one embodiment of this specification is shown.

[0234] The computing device 900 includes a memory 910 and a processor 920; the memory 910 is used to store computer programs / instructions, and the processor 920 is used to execute the computer programs / instructions, which, when executed by the processor 920, implement the steps of the data processing method and the agent training method.

[0235] In one or more embodiments of this specification, the computing device can be understood as an integrated smart terminal, including but not limited to a server, desktop computer, PC (Personal Computer), all-in-one model machine, mobile phone, tablet computer or other portable smart terminal, etc., and the computing device may have the model described in the above embodiments of this application pre-installed.

[0236] Specifically, this computing device can pre-install various types of models, including but not limited to models in natural language processing, visual processing, speech processing, code processing, and multimodal task processing, thus providing diverse model selection. In different product forms, this computing device can support one or more model usage methods, including but not limited to model training, model invocation, model fine-tuning, model deployment, model inference, and application. In some product forms, this computing device also supports model management, including but not limited to multi-type model management (supporting the management of discriminative, generative, and other model types), model version control (supporting the control of different model versions), and model evaluation (evaluating model performance and effectiveness based on model evaluation tools). In other product forms, this computing device can also create applications based on models, providing API (Application Programming Interface) calling capabilities. Users can call models into created applications through the API interface, and application management tools are also provided to manage and monitor the applications.

[0237] Furthermore, the computing device can also include data management (supporting the creation and management of model tuning datasets), a training center (providing abundant training resources to help users learn and master AI (Artificial Intelligence) technology), and basic control capabilities (providing enterprise-level basic control capabilities to ensure the security and efficient operation of the system). Through the above functions, it provides a comprehensive and integrated device for AI development, training, deployment, and application.

[0238] Figure 10 A structural block diagram of an electronic device 1000 provided according to one embodiment of this specification is shown.

[0239] A memory 1010 and a processor 1020 are connected via a bus 1030; the memory 1010 is used to store computer programs / instructions, and the processor 1020 is used to execute the computer programs / instructions, which, when executed by the processor 1020, implement the steps of the method.

[0240] Specifically, the components of the electronic device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.

[0241] Electronic device 1000 also includes access device 1040, which enables electronic device 1000 to communicate via one or more networks 1060. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. Access device 1040 may include one or more of any type of wired or wireless network interface (e.g., network interface card (NIC)), such as an IEEE 1002.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0242] In one embodiment of this specification, the above-described components of the electronic device 1000 and Figure 10 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 10 The block diagram of the electronic device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0243] Electronic device 1000 can be any type of stationary or mobile electronic device, including mobile computers or mobile electronic devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable electronic devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary electronic devices such as desktop computers or personal computers (PCs). Electronic device 1000 can also be a mobile or stationary server.

[0244] The above is an illustrative scheme of an electronic device according to this embodiment. It should be noted that the technical solution of this electronic device belongs to the same concept as the technical solutions of the above-mentioned agent training method, data processing method, and question answering method. For details not described in detail in the technical solution of the electronic device, please refer to the descriptions of the technical solutions of the above-mentioned agent training method, data processing method, and question answering method.

[0245] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described agent training method, data processing method, and question-answering method.

[0246] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the above-described agent training method, data processing method, and question answering method. For details not described in detail in the technical solution of the storage medium, please refer to the descriptions of the technical solutions of the above-described agent training method, data processing method, and question answering method.

[0247] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described agent training method, data processing method, and question-answering method.

[0248] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the above-mentioned agent training method, data processing method, and question answering method. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the above-mentioned agent training method, data processing method, and question answering method.

[0249] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0250] The computer program / instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0251] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0252] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0253] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An agent training method, comprising: training an initial agent using multi-turn reference data and corresponding reference results, to obtain an updated agent, wherein the reference data is reference questions input by an interactive party in a dialogue scenario, and the reference results are reference replies corresponding to the reference questions; obtaining multi-turn exploration data and corresponding exploration results according to the updated agent, wherein the first-turn exploration data is the same as the first-turn reference data, the exploration data is exploration questions input by the interactive party, and the exploration results are exploration replies generated by the updated agent for the exploration questions; determining positive replies and negative replies according to annotation information of the multi-turn reference replies and the multi-turn exploration replies, wherein a guide reply included in the reference replies and / or the exploration replies is a positive reply, and the guide reply is used to guide the interactive party to output data containing atomic facts; training the updated agent according to a target question, the positive replies, and the negative replies, to obtain a target agent, wherein the target question is a reference question or an exploration question.

2. The agent training method of claim 1, before the step of training the initial agent using the multi-turn reference data and the corresponding reference results to obtain the updated agent, further comprising: determining reference data and fact labels corresponding to the reference data, wherein the reference data is reference data of a first turn in the multi-turn reference data, the multi-turn reference data corresponds to a domain event containing one or more atomic facts, and the fact labels are mapping labels corresponding to the atomic facts; generating reference results corresponding to the reference data according to the reference data and the fact labels, wherein the reference results are guide results or answer results; updating the reference data to response reference data corresponding to the reference results, iteratively performing the steps of determining the reference data and the fact labels corresponding to the reference data, to obtain the multi-turn reference data and the corresponding reference results.

3. The agent training method of claim 2, wherein the step of determining the reference data and the fact labels corresponding to the reference data comprises: determining the reference data and a domain event corresponding to the reference data; determining the fact labels corresponding to the reference data from a label set corresponding to the domain event.

4. The agent training method of claim 1, wherein the step of obtaining the multi-turn exploration data and the corresponding exploration results according to the updated agent comprises: inputting exploration data to the updated agent to obtain exploration results output by the updated agent, wherein the exploration data is exploration data of a first turn in the multi-turn exploration data, and the multi-turn exploration data corresponds to a domain event containing one or more atomic facts; updating the exploration data to response exploration data corresponding to the exploration results output by the interactive party, iteratively performing the step of inputting the exploration data to the updated agent, to obtain the multi-turn exploration data and the corresponding exploration results. 5.The method of claim 1, wherein the determining the positive reply and the negative reply based on the labeled information of the multi-turn reference reply and the multi-turn exploration reply comprises: adding the labeled information to the multi-turn reference reply and the multi-turn exploration reply based on atomic fact sets of domain events corresponding to the multi-turn reference question and the multi-turn exploration question; determining the reference reply and / or the exploration reply corresponding to the labeled information as the positive reply when the labeled information is positive; and determining the reference reply and / or the exploration reply corresponding to the labeled information as the negative reply when the labeled information is negative. 6.The method of claim 5, wherein the adding the labeled information to the multi-turn reference reply and the multi-turn exploration reply based on atomic fact sets of domain events corresponding to the multi-turn reference question and the multi-turn exploration question comprises: determining a target reference atomic fact contained in the multi-turn reference question and a target exploration atomic fact contained in the multi-turn reference question from the atomic fact sets; and determining a target atomic fact set based on the target reference atomic fact and the target exploration atomic fact, and adding the labeled information to the multi-turn reference reply and the multi-turn exploration reply based on the target atomic fact set. 7.The method of claim 6, wherein the determining a target reference atomic fact contained in the multi-turn reference question and a target exploration atomic fact contained in the multi-turn reference question from the atomic fact sets comprises: determining a reference atomic fact set corresponding to the multi-turn reference question based on the multi-turn reference question and the atomic fact sets; determining an exploration atomic fact set corresponding to the multi-turn exploration question based on the multi-turn exploration question and the atomic fact sets, wherein the reference atomic fact set and the exploration atomic fact set are subsets of the atomic fact sets; and determining the target reference atomic fact from the reference atomic fact set and the target exploration atomic fact from the exploration atomic fact set based on a difference between the reference atomic fact set and the exploration atomic fact set, wherein the target reference atomic fact belongs to the reference atomic fact set but not to the exploration atomic fact set, and the target exploration atomic fact belongs to the exploration atomic fact set but not to the reference atomic fact set. 8.The method of claim 6, wherein the adding the labeled information to the multi-turn reference reply and the multi-turn exploration reply based on the target atomic fact set comprises: determining a target reference reply from the multi-turn reference reply and a target exploration reply from the multi-turn exploration reply; and adding the labeled information to the target reference reply and the target exploration reply based on the target atomic fact set. ​ ​ ​ ​ ​ ​ ​ ​ ​ determining a response reference question corresponding to the target reference reply from the multiple rounds of reference questions, adding positive label information to the target reference reply in a case where a response atomic fact contained in the response reference question belongs to the target atomic fact set, or adding negative label information to the target reference reply in a case where the response atomic fact does not belong to the target atomic fact set; determining a response exploration question corresponding to the target exploration reply from the multiple rounds of exploration questions, adding positive label information to the target exploration reply in a case where a response atomic fact contained in the response exploration question belongs to the target atomic fact set, or adding negative label information to the target exploration reply in a case where the response atomic fact does not belong to the target atomic fact set; continuing to perform the steps of determining a target reference reply from the multiple rounds of reference replies and determining a target exploration reply from the multiple rounds of exploration replies until the label information is added to the multiple rounds of reference replies and the multiple rounds of exploration replies.

9. The agent training method of any one of claims 1-8, wherein the training the updated agent according to the target question, the positive reply, and the negative reply to obtain a target agent comprises: constructing a first question pair corresponding to the positive reply based on a target question corresponding to the positive reply and an off-track negative reply determined from the negative reply, wherein the positive reply and the off-track negative reply are from different sources; constructing a second question pair corresponding to the negative reply based on a target question corresponding to the negative reply and an off-track positive reply determined from the positive reply; training the updated agent according to the first question pair corresponding to the positive reply and the second question pair corresponding to the negative reply to obtain the target agent.

10. A question and answer method, comprising: inputting an initial question to a target agent to obtain an initial reply output by the target agent, wherein the target agent is obtained according to the agent training method of any one of claims 1-9; in response to a response question corresponding to the initial reply, determining the response question as the initial question, and continuing to perform the step of inputting an initial question to a target agent until a stop condition is met.

11. A question and answer method applied to a question and answer platform, wherein a target agent is deployed on the question and answer platform, and the method comprises: in response to an inquiry request sent by a client, obtaining an initial question carried in the inquiry request; inputting the initial question to the target agent to obtain an initial reply output by the target agent, and returning the initial reply to the client, wherein the target agent is obtained according to the agent training method of any one of claims 1-9; in response to a follow-up request sent by the client corresponding to the initial reply, obtaining a response question corresponding to the initial reply, determining the response question as the initial question, and continuing to perform the step of inputting the initial question to the target agent until a stop condition is met.

12. A computing device, comprising: a memory and a processor; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-11.

13. An electronic device comprising: a memory and a processor connected by a bus; The memory is configured to store computer programs / instructions, and the processor is configured to execute the computer programs / instructions, and the computer programs / instructions, when executed by the processor, implement the steps of the method according to any one of claims 1-11.

14. A computer readable storage medium storing computer programs / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1-11.

15. A computer program product comprising computer programs / instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1-11.

Images