Evaluation method and device for the clarification ability of conversational agents
By using hidden constraint sets and predefined rules to generate response text in conversational agent evaluation, the problems of non-reproducibility and incomparability of evaluation results are solved, and the quantifiable evaluation of the agent's clarification ability and the utilization of traceable constraint information are realized.
Patent Information
- Application Number
- CN202611381656.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-07
- Publication Date
- 2026-11-10
Smart Images

Figure CN123068299A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to the field of computer software technology, and more particularly to a method and apparatus for evaluating the clarification capabilities of a conversational agent. Background Technology
[0002] With the rapid development of large language model technology, conversational agents are widely used in customer service, e-commerce, and other fields. These agents need to proactively clarify ambiguous intentions when user requests are incomplete; therefore, evaluating their clarification ability is a crucial dimension for assessing the quality of agent service. In related technologies, to evaluate the agent's clarification ability, large language models are typically used to simulate users, or real users are employed in the evaluation.
[0003] The text generation process of large language models and the responses of real users are both random. When faced with the same clarification question, even if the large language model inputs the same prompt words, the responses generated by the model and real users will be inconsistent. This makes it impossible to reproduce the evaluation results of the same agent multiple times, and the evaluation results between different agents lack comparability. Summary of the Invention
[0004] This specification provides one or more embodiments of a method and apparatus for evaluating the clarification ability of a conversational agent, thereby solving the problems of non-reproducible results and incomparable evaluation conclusions in existing evaluation methods.
[0005] In a first aspect, one or more embodiments of this specification provide a method for evaluating the clarification ability of a conversational agent, comprising: acquiring a first instruction and a set of hidden constraints, wherein the first instruction is used to instruct the agent under test to perform a target operation and does not contain at least one hidden constraint text required to perform the target operation, the set of hidden constraints includes at least one hidden constraint text, and the set of hidden constraints is invisible to the agent under test; sending the first instruction to the agent under test, and conducting a multi-turn dialogue with the agent under test based on the first instruction and the set of hidden constraints; and obtaining an evaluation result of the clarification ability of the agent under test based on the dialogue content of the multi-turn dialogue.
[0006] In one possible implementation, a multi-turn dialogue is conducted with the agent under test based on a first instruction and a set of hidden constraints, including: for each question initiated by the agent under test, searching for target hidden constraint text that matches the question from the set of hidden constraints according to predefined rules, wherein the set of hidden constraints matches the target operation indicated by the first instruction; if the target hidden constraint text is found, generating a response text for the question based on the target hidden constraint text and sending it to the agent under test; if the target hidden constraint text is not found, generating a response text for the question based on the question type according to a pre-configured strategy and sending it to the agent under test.
[0007] In one possible implementation, searching for target hidden constraint text matching the question from the set of hidden constraints according to predefined rules includes: classifying the question to obtain the question type; searching for target hidden constraint text matching the question from the set of hidden constraints according to the search method matching the question type; and generating response text for the question based on the target hidden constraint text, including: generating response text based on the response strategy corresponding to the pre-configured behavior profile and the target hidden constraint text, wherein the behavior profile is used to limit the number of hidden constraint texts that are allowed to be revealed in the response text.
[0008] In one possible implementation, the questions are classified to obtain question types, including: if the matching degree between the question text and an irrelevant text set exceeds a first matching degree threshold, the question is determined to be an irrelevant question, wherein the irrelevant text set is used to represent topics unrelated to the target operation; if the matching degree between the question text and an open text set exceeds a second matching degree threshold, the question is determined to be an open question, wherein the open text set includes a list of open question keywords; if the question text matches at least one attribute category in an attribute category set, the question is determined to be a category question, wherein each hidden constraint text in the hidden constraint set belongs to at least one attribute category in the attribute category set; if the number of attribute categories matched by the category question exceeds a preset number threshold, or if the question contains a conjunction, the category question is determined to be a broad question, wherein the conjunction is used to connect multiple attribute types in parallel.
[0009] In one possible implementation, when the question type is a category question, searching for target hidden constraint text that matches the question from the hidden constraint set includes: performing string matching between the question text and each hidden constraint text in the hidden constraint set; if the question text contains a string that matches any hidden constraint text, determining the hidden constraint text that matches the question text as the target hidden constraint text; if the question text does not contain a string that matches any hidden constraint text, determining the attribute category that matches the question text as the target attribute category, and filtering candidate hidden constraint texts belonging to the target attribute category from the hidden constraint set; performing word segmentation on the question text and at least one candidate hidden constraint text respectively, and calculating the overlap score between the word segmentation results; and determining the candidate hidden constraint text whose overlap score meets a preset scoring condition as the target hidden constraint text.
[0010] In one possible implementation, when the behavior profile is the first profile, the response text is generated based on the response strategy corresponding to the pre-configured behavior profile and the target hidden constraint text, including: (1) when the question type is an open question, when the number of hidden constraint texts in the hidden constraint set is not greater than a first preset number, the response text is generated based on all hidden constraint texts; when the number of hidden constraint texts is greater than the first preset number, the first N hidden constraint texts are selected according to a preset sorting to generate the response text, and a first prompt message is added to the response text, where N is an integer not less than 1, and the first prompt message is used to instruct the tested agent to recommend and determine the requirements corresponding to the unrevealed hidden constraint texts; (2) when the question type is a category question, the response text is generated based on the target hidden constraint text and the matched attribute category.
[0011] In one possible implementation, when the behavior profile is the second profile, the response text is generated based on the response strategy corresponding to the pre-configured behavior profile and the target hidden constraint text, including: (1) when the question type is an open question, a hidden constraint text is selected according to a preset sorting to generate the response text, and a second prompt message is added to the response text, wherein the second prompt message is used to guide the tested agent to ask questions separately for other attribute categories; (2) when the question type is a category question, the response text is generated based on the target hidden constraint text and the matched attribute category.
[0012] In one possible implementation, when the behavior profile is a third profile, the response text is generated based on the response strategy and target hidden constraint text corresponding to the pre-configured behavior profile, including: (1) when the question type is an open question, the response text is generated based on all hidden constraint texts in the hidden constraint set, and no additional prompt information is added to the response text; (2) when the question type is a category question, the response text is generated based on the target hidden constraint text and the matched attribute category.
[0013] In one possible implementation, if no target hidden constraint text is found, a response text is generated based on the question type according to a pre-configured strategy. This includes: if the question type is an irrelevant question and the behavior profile is a first profile or a third profile, a generalized response text is generated according to a pre-configured irrelevant question handling strategy, indicating that there is no additional requirement for the target operation; if the question type is an irrelevant question and the behavior profile is a second profile, a negative feedback text is generated according to a pre-configured irrelevant question handling strategy, indicating negative feedback to the question; if the question type is a category question and the behavior profile is a second profile, a confused response text is generated, indicating that the question cannot be understood; and if the question type is a category question and the behavior profile is a first profile or a third profile, a deflection response text is generated, indicating that there is no preference for the attribute category involved in the question.
[0014] In one possible implementation, after generating the response text to the question, the method further includes: performing string matching between the response text and each hidden constraint text in the hidden constraint set; identifying the hidden constraint text contained in the response text as the recovered hidden constraint text, and adding the recovered hidden constraint text to the recovered constraint set, wherein the recovered constraint set is used to accumulate and record the recovered hidden constraint text during multi-turn dialogue.
[0015] In one possible implementation, before performing string matching between the response text and each hidden constraint text in the hidden constraint set, the method further includes: normalizing the response text and / or the hidden constraint text, the normalization process including at least one of the following: normalizing the text for capitalization; replacing synonyms in the response text with standard expressions based on a predefined synonym mapping table, wherein the synonym mapping table includes at least one of a color synonym mapping table, a size normalization mapping table, a material synonym mapping table, and a brand synonym mapping table.
[0016] In one possible implementation, a multi-turn dialogue is conducted with the tested agent based on a first instruction and a set of hidden constraints. This includes: when the tested agent performs a question action and a preset trigger condition is met, replacing the question action with a search action, executing the search action, and returning the search results to the tested agent. The preset trigger condition includes at least one of the following: the cumulative number of questions performed by the question action reaches a preset limit; the question action is repeated with a historical question action in the multi-turn dialogue; the most recent response text in the multi-turn dialogue is an inoperable response text, which does not contain executable constraint information.
[0017] In one possible implementation, performing a search action includes: constructing a search query based on a first instruction and a set of restored constraints, the search query including the first instruction and a preset number of restored hidden constraint texts not included in the first instruction; and performing a search based on the search query.
[0018] In one possible implementation, after conducting multiple rounds of dialogue with the agent under test based on the first instruction and the hidden constraint set, the method further includes: obtaining the description text of the target object indicated by the target operation when the agent under test completes the target operation; detecting the overlap between the description text and the hidden constraint set based on a predefined thesaurus mapping table and word boundary regular expressions; determining the task execution result of the agent under test based on the detection result, and using the task execution result to obtain the clarification ability evaluation result.
[0019] In one possible implementation, the clarification ability evaluation result of the tested agent is obtained based on the dialogue content of a multi-turn dialogue. This includes: classifying the types of questions initiated by the tested agent in the multi-turn dialogue and statistically obtaining the question type distribution. The question types include specific questions, categorical questions, open-ended questions, and other types. Specific questions are used to inquire about the specific values of constraints, categorical questions are used to inquire about predefined attribute categories, and open-ended questions are used to inquire about needs in a general way. The ratio of the number of hidden constraint texts in the recovered constraint set to the total number of hidden constraint texts in the hidden constraint set is determined as the attribute recovery rate. The constraint integration rate is determined based on the number of hidden constraint texts in the recovered constraint set included in the search query of the tested agent and the total number of hidden constraint texts in the recovered constraint set. The clarification ability evaluation result is obtained based on the question type distribution, the attribute recovery rate, and / or the constraint integration rate.
[0020] In one possible implementation, the types of questions initiated by the agent under test in a multi-turn dialogue are classified, including: classifying each question according to a predefined classification rule to obtain a first classification result; classifying the questions according to a large language model to obtain a second classification result; calculating the consistency rate between the first classification result and the second classification result, and verifying and updating the question type distribution based on the consistency rate.
[0021] In one possible implementation, the evaluation result of the clarification ability of the tested agent is obtained based on the dialogue content of the multi-turn dialogue, and further includes: adjusting the behavioral profile to at least two different behavioral profiles, conducting multi-turn dialogues with the tested agent under the at least two different behavioral profiles respectively, and obtaining the task execution result corresponding to each behavioral profile; determining the profile sensitivity of the tested agent's clarification ability based on the difference between the task execution results corresponding to the at least two different behavioral profiles, and obtaining the clarification ability evaluation result based on the profile sensitivity.
[0022] Secondly, one or more embodiments of this specification provide an evaluation device for the clarification ability of a conversational intelligent agent, comprising: an acquisition module, configured to acquire a first instruction and a set of hidden constraints, wherein the first instruction is used to instruct the intelligent agent under test to perform a target operation and does not contain at least one hidden constraint text required to perform the target operation, the set of hidden constraints includes at least one hidden constraint text, and the set of hidden constraints is invisible to the intelligent agent under test; a dialogue module, configured to send the first instruction to the intelligent agent under test and conduct multi-turn dialogue with the intelligent agent under test based on the first instruction and the set of hidden constraints; and a result determination module, configured to obtain an evaluation result of the clarification ability of the intelligent agent under test based on the dialogue content of the multi-turn dialogue.
[0023] In one possible implementation, the dialogue module is further configured to: for each question initiated by the agent under test, search for a target hidden constraint text matching the question from a set of hidden constraints according to predefined rules, wherein the set of hidden constraints matches the target operation indicated by the first instruction; if a target hidden constraint text is found, generate a response text for the question based on the target hidden constraint text and send it to the agent under test; if no target hidden constraint text is found, generate a response text for the question based on the question type according to a pre-configured strategy and send it to the agent under test.
[0024] In one possible implementation, the dialogue module is further configured to: classify the questions to obtain the question types; search for target hidden constraint text that matches the question from the hidden constraint set according to a search method that matches the question type; and generate response text based on the response strategy corresponding to the pre-configured behavior profile and the target hidden constraint text, wherein the behavior profile is used to limit the number of hidden constraint texts that are allowed to be revealed in the response text.
[0025] In one possible implementation, the dialogue module is further configured to: determine the question as an irrelevant question when the matching degree between the question text and the irrelevant text set exceeds a first matching degree threshold, wherein the irrelevant text set is used to represent topics unrelated to the target operation; determine the question as an open question when the matching degree between the question text and the open text set exceeds a second matching degree threshold, wherein the open text set includes a list of open question keywords; determine the question as a category question when the question text matches at least one attribute category in the attribute category set, wherein each hidden constraint text in the hidden constraint set belongs to at least one attribute category in the attribute category set; and determine the category question as a broad question when the number of attribute categories matched by the category question exceeds a preset number threshold, or when the question contains a conjunction, wherein the conjunction is used to connect multiple attribute types in parallel.
[0026] In one possible implementation, the dialogue module is further configured to: perform string matching between the question text and each hidden constraint text in the hidden constraint set; if the question text contains a string that matches any hidden constraint text, determine the hidden constraint text that matches the question text as the target hidden constraint text; if the question text does not contain a string that matches any hidden constraint text, determine the attribute category that matches the question text as the target attribute category, and filter candidate hidden constraint texts belonging to the target attribute category from the hidden constraint set; perform word segmentation on the question text and at least one candidate hidden constraint text respectively, and calculate the overlap score between the word segmentation results; determine the candidate hidden constraint text whose overlap score meets the preset scoring conditions as the target hidden constraint text.
[0027] In one possible implementation, the dialogue module is further configured to: (1) when the question type is an open question, generate a response text based on all hidden constraint texts when the number of hidden constraint texts in the hidden constraint set is not greater than a first preset number; when the number of hidden constraint texts is greater than the first preset number, select the first N hidden constraint texts according to a preset sorting to generate a response text, and add a first prompt message to the response text, wherein N is an integer not less than 1, and the first prompt message is used to instruct the tested agent to recommend and determine the requirements corresponding to the unrevealed hidden constraint texts; (2) when the question type is a category question, generate a response text based on the target hidden constraint text and the matched attribute category.
[0028] In one possible implementation, the dialogue module is further configured to: (1) select a hidden constraint text according to a preset sorting to generate a response text when the question type is an open question, and add a second prompt message to the response text, wherein the second prompt message is used to guide the tested agent to ask questions separately for other attribute categories; (2) generate a response text based on the target hidden constraint text and the matched attribute category when the question type is a category question.
[0029] In one possible implementation, the dialogue module is further configured to: (1) generate a response text based on all hidden constraint texts in the hidden constraint set when the question type is an open question, and the response text does not include any additional prompt information; (2) generate a response text based on the target hidden constraint text and the matching attribute category when the question type is a category question.
[0030] In one possible implementation, the dialogue module is further configured to: generate a generalized response text according to a pre-configured irrelevant question handling strategy when the question type is an irrelevant question and the behavior profile is a first profile or a third profile; the generalized response text indicates that there is no additional requirement for the target operation when the question type is an irrelevant question and the behavior profile is a second profile; generate a negative feedback text according to a pre-configured irrelevant question handling strategy when the question type is an irrelevant question and the behavior profile is a second profile; generate a confused response text when the question type is a category question and the behavior profile is a second profile; and generate a deflection response text when the question type is a category question and the behavior profile is a first profile or a third profile. The deflection response text indicates that there is no preference for the attribute category involved in the question.
[0031] In one possible implementation, the dialogue module is further configured to: perform string matching between the response text and each hidden constraint text in the hidden constraint set; determine the hidden constraint text contained in the response text as the recovered hidden constraint text, and add the recovered hidden constraint text to the recovered constraint set, wherein the recovered constraint set is used to accumulate and record the recovered hidden constraint text during multi-turn dialogue.
[0032] In one possible implementation, the dialogue module is further configured to: normalize the response text and / or hidden constraint text, the normalization process including at least one of the following: normalizing the text for capitalization; and replacing synonyms in the response text with standard expressions based on a predefined synonym mapping table, wherein the synonym mapping table includes at least one of a color synonym mapping table, a size normalization mapping table, a material synonym mapping table, and a brand synonym mapping table.
[0033] In one possible implementation, the dialogue module is further configured to: replace the questioning action with a search action when the agent under test performs a questioning action and a preset triggering condition is met, execute the search action, and return the search results to the agent under test; wherein the preset triggering condition includes at least one of the following: the cumulative number of questions performed by the questioning action reaches a preset limit; the questioning action is repeated with a historical questioning action in a multi-turn dialogue; the most recent reply text in the multi-turn dialogue is an inoperable reply text, and the inoperable reply text does not contain executable constraint information.
[0034] In one possible implementation, the dialogue module is further configured to: construct a search query based on the first instruction and the set of restored constraints, the search query including the first instruction and a preset number of restored hidden constraint texts not included in the first instruction; and perform a search based on the search query.
[0035] In one possible implementation, the dialogue module is further configured to: obtain the description text of the target object indicated by the target operation when the tested agent completes the target operation; detect the overlap between the description text and the hidden constraint set based on a predefined synonym mapping table and word boundary regular expressions; determine the task execution result of the tested agent based on the detection result, and the task execution result is used to obtain the clarification ability evaluation result.
[0036] In one possible implementation, the result determination module is further configured to: classify the types of questions initiated by the tested agent in a multi-turn dialogue, and statistically obtain the question type distribution, wherein the question types include specific questions, categorical questions, open-ended questions, and other types of questions. Specific questions are used to inquire about the specific values of constraints, categorical questions are used to inquire about predefined attribute categories, and open-ended questions are used to inquire about requirements in general terms; determine the attribute recovery rate by the ratio of the number of hidden constraint texts in the recovered constraint set to the total number of hidden constraint texts in the hidden constraint set; determine the constraint integration rate based on the number of hidden constraint texts in the recovered constraint set included in the search query of the tested agent; and obtain the clarification ability evaluation result based on the question type distribution, attribute recovery rate, and / or constraint integration rate.
[0037] In one possible implementation, the result determination module is further configured to: classify each question according to a predefined classification rule to obtain a first classification result; classify the questions according to a large language model to obtain a second classification result; calculate the consistency rate between the first classification result and the second classification result, and verify and update the question type distribution based on the consistency rate.
[0038] In one possible implementation, the result determination module is further configured to: adjust the behavior profile to at least two different behavior profiles, conduct multiple rounds of dialogue with the tested agent under at least two different behavior profiles, and obtain the task execution results corresponding to each behavior profile; determine the profile sensitivity of the tested agent's clarification ability based on the differences between the task execution results corresponding to at least two different behavior profiles, and obtain the clarification ability evaluation result based on the profile sensitivity.
[0039] Thirdly, one or more embodiments of this specification also provide an electronic device, which includes a memory and a processor; the memory is used to store a computer program product; the processor is used to execute the computer program product stored in the memory, and when the computer program product is executed, it implements the method of the first or second aspect described above.
[0040] Fourthly, one or more embodiments of this specification also provide a computer-readable storage medium storing computer program instructions that, when executed, implement the methods of the first or second aspect described above.
[0041] Based on the implementation methods provided in the above aspects, one or more embodiments of this specification can be further combined to provide more implementation methods.
[0042] By employing the above method, the evaluation system sends only the first instruction carrying the target operation to the tested agent, and makes the hidden constraint set invisible to the tested agent, thus isolating the data between the tested agent and the actual needs. In this way, the tested agent can only complete the constraints by clarifying the questions, thereby recreating the interaction scenario where the user's needs are not fully expressed under controlled conditions. During multi-turn dialogues, the evaluation system's responses are limited to the predefined hidden constraint set, avoiding information leakage caused by simulating users deviating from preset preferences. Furthermore, the responses generated based on predefined rules are deterministic. Under the same task configuration (such as the same hidden constraint set and the same target operation), the interaction process and results of multiple evaluations are consistent, making the evaluation results comparable between different tested agents and between different versions of the same tested agent. This solves the problems of non-reproducible results and incomparable evaluation conclusions in existing evaluation methods. Attached Figure Description
[0043] To more clearly illustrate the technical solutions of one or more embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of one or more embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0044] Figure 1 A system schematic diagram of a method for evaluating the clarification ability of a conversational agent provided in one or more embodiments of this specification; Figure 2 A flowchart illustrating a method for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification. Figure 3 A schematic diagram illustrating the construction of an evaluation task for an evaluation method of the clarification ability of a conversational agent provided in one or more embodiments of this specification; Figure 4 A schematic diagram of the architecture of a user simulator for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification. Figure 5A schematic diagram of a user simulator for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; Figure 6 A schematic diagram showing the architectural details of a method for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification. Figure 7 A schematic diagram illustrating a method for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; Figure 8 A structural block diagram of an evaluation device for the clarification ability of a conversational agent provided in one or more embodiments of this specification; Figure 9 This is a structural block diagram of an electronic device provided for one or more embodiments of this specification. Detailed Implementation
[0045] The present specification describes one or more embodiments in further detail below with reference to the accompanying drawings and examples. Through these descriptions, the features and advantages of one or more embodiments of the present specification will become clearer and more apparent.
[0046] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.
[0047] Furthermore, the technical features described below in one or more embodiments of this specification may be combined with each other as long as they do not conflict with each other.
[0048] It should be noted that the acquisition and processing of user data involved in the methods described in this manual strictly adhere to the principles of legality, legitimacy, and necessity, and the entire process has been authorized and consented to by the user. Before collecting user data, the platform has informed the user of the purpose, scope, and method of data use through a clear and easy-to-understand authorization agreement, and the user can choose whether to authorize and the specific scope of authorization.
[0049] With the rapid development of large language model technology, conversational agents are widely used in e-commerce guidance, job search assistance, and government service scenarios. Taking Alipay's job assistant as an example, after a user requests "help me find a job," the assistant needs to proactively ask questions when the request is incomplete, clarifying preferences such as work location and salary range, and then searching and recommending jobs based on the clarification results. In such scenarios, the agent's clarification ability directly determines the service quality. To evaluate the agent's clarification ability, it is necessary to conduct an evaluation before the agent goes live: the evaluation system simulates multiple rounds of dialogue between the user and the agent, and evaluates the agent's clarification ability based on the dialogue process.
[0050] As described in the background section, in related technologies, the clarification ability of an agent is typically evaluated by real users or by a large language model simulating a user. This approach suffers from several drawbacks. Real users exhibit individual differences and unpredictable levels of cooperation in their response styles, leading to high evaluation costs and unreproducible results. Furthermore, the text generation process of large language models is random; even with the same prompts and question sequences, multiple generated responses will differ, making it impossible to reproduce the evaluation results of the same agent and resulting in a lack of comparability between different agents. Moreover, large language models may deviate from preset user preferences when generating responses, producing content outside of these preferences. This can cause the agent to execute subsequent actions based on incorrect responses, thus distorting the evaluation of the agent's clarification ability.
[0051] To address the aforementioned problems, one or more embodiments of this specification provide a method for evaluating the clarification ability of a conversational agent. The method is executed by an evaluation system: acquiring a first instruction and a set of hidden constraints, wherein the first instruction instructs the agent under test to perform a target operation but does not contain at least one hidden constraint text, and the set of hidden constraints is invisible to the agent under test; sending the first instruction to the agent under test, and engaging in multi-turn dialogue with the agent under test based on the first instruction and the set of hidden constraints, during which the evaluation system determines the target hidden constraint text matching the question from the set of hidden constraints according to predefined rules and generates a response; and obtaining the evaluation result of the agent under test's clarification ability based on the content of the multi-turn dialogue.
[0052] Through the above methods, the evaluation system's response content is limited to a predefined set of hidden constraints, avoiding information leakage caused by simulated users acting freely outside of preset preferences. Responses generated based on predefined rules are deterministic, with the interaction process and results being completely consistent under the same task configuration and question sequence, making the evaluation results reproducible and comparable. Furthermore, the evaluation results are derived from complete dialogue content, quantifying the tested agent's ability to acquire constraint information and tracking the utilization of constraint information in subsequent execution, thus achieving a diagnostic evaluation of clarification capabilities.
[0053] For ease of understanding, the application scenarios of the technical solutions provided in one or more embodiments of this specification are described below.
[0054] Optionally, in one example, the clarification ability of an e-commerce shopping guide agent can be evaluated, such as an e-commerce platform conducting a pre-launch evaluation of the agent's clarification ability. The evaluation system receives an evaluation task, where the first instruction is "Buy me a shirt," and the hidden constraint set is {"blue," "pure cotton," "under 200 yuan"}, which is invisible to the shopping guide agent. The evaluation system sends the first instruction to the shopping guide agent, which asks, "What are your color preferences?" The evaluation system categorizes this question as a category question. After a string matching error, it determines the corresponding color category based on a predefined attribute category keyword table. It then filters "blue," belonging to the color category, from the hidden constraint set. After word segmentation and overlap scoring verification, it is determined as the target hidden constraint text and generates the response text "I prefer blue" according to the first profile, returning it to the shopping guide agent. Simultaneously, "blue" is added to the recovery constraint set. After multiple rounds of dialogue, "pure cotton" and "under 200 yuan" are gradually recovered. When the shopping guide agent initiates a search, the evaluation system checks whether the constraints in the restored constraint set are incorporated into its search query. After the shopping guide agent completes the purchase, the evaluation system obtains the description text of the purchased product and checks whether it satisfies all hidden constraints to determine the task execution result, thereby obtaining the evaluation result of the shopping guide agent's clarification ability.
[0055] Optionally, in one example, the clarification ability of the employment assistant agent can be evaluated. For instance, if a large number of users search for jobs through the Alipay employment assistant (agent), and user requests are generally incomplete, the platform needs to evaluate the employment assistant's clarification ability. The evaluation system receives an evaluation task, where the first instruction is "Help me find a job," and the hidden constraint set is {"City A," "Monthly salary no less than 8000 yuan," "Internet industry"}. After receiving the first instruction, the employment assistant asks, "Do you have any requirements regarding the work location?" The evaluation system identifies this question as a category question, determines its corresponding location category, generates the reply text "Hopefully in City A" and returns it, while simultaneously adding "City A" to the restored constraint set. Subsequently, with two constraints restored, the employment assistant asks a broader question, "Do you have any other requirements?" The evaluation system determines that a preset trigger condition is met, replaces the question action with a search action, constructs a search query based on the first instruction and constraints in the restored constraint set not included in the first instruction, executes it, and returns the job search results to the employment assistant. After the employment assistant completes the job recommendation, the evaluation system obtains the clarification ability evaluation results of the employment assistant based on factors such as the number of constraints restored.
[0056] Optionally, in one example, the clarification ability of the government affairs assistant agent can be evaluated. For instance, if a user applies for a residence permit through the Alipay government affairs assistant, the platform needs to evaluate the assistant's clarification ability when dealing with cautious citizens. The evaluation system receives the evaluation task, where the first instruction is "I want to apply for a residence permit," the hidden constraint set is {"non-local," "residing for 6 months," "Area B"}, and the behavioral profile is configured as the second profile. The government affairs assistant initiates an open question, "What information do you need to supplement?" Following the response strategy of the second profile, the evaluation system selects only one hidden constraint text according to a preset sorting to generate the response text "I am non-local," with added guiding prompts to guide the assistant to ask questions about other conditions separately. The assistant then initiates a specific question, "Have you resided in this area for 6 months?" The evaluation system matches the string and finds "residing for 6 months," generating a precise response. If the assistant continues to initiate broad questions and the number of questions reaches a preset limit, the evaluation system replaces subsequent questioning actions with search actions, directly returning the search results for the application guidance to the assistant, avoiding an excessive clarification loop in the dialogue. After the task is completed, the evaluation system analyzes the distribution of question types in the questions asked by the government affairs assistant, and combines the task execution results to obtain the evaluation results of the government affairs assistant's clarification ability in cautious user scenarios.
[0057] The above use cases are merely examples. Any scenario related to evaluating the clarification capabilities of conversational agents can utilize the solutions of one or more embodiments of this specification. This embodiment does not impose any limitations on these applications.
[0058] As an optional implementation, the above-described method for evaluating the clarification ability of conversational agents can be applied, but is not limited to, to applications such as... Figure 1 The system shown is an evaluation system for the clarification capabilities of a conversational agent. This evaluation system may include, but is not limited to, a terminal device 102, a database server 104, a first server cluster 106, a network 112, and a second server cluster 114.
[0059] like Figure 1 In the illustrated embodiment, terminal device 102 is used to provide evaluation task data, such as a mobile phone, laptop, or desktop computer. Terminal device 102 can serve as a configuration entry point for task data; for example, evaluators can input or upload evaluation task data (first instructions and hidden constraint set) through the terminal device.
[0060] Database server 104 is a server used for offline storage of evaluation task data. It stores a pre-built task library, which includes the first instruction and hidden constraint set of each evaluation task, and may also include metadata such as difficulty level and data traceability identifier.
[0061] The first server cluster 106 is the server cluster on the side of the intelligent agent under test, including a database 108 and a first processing engine 110. The first processing engine 110 is used to run the intelligent agent under test, receive first instructions, output actions (question actions, search actions, etc.), and receive responses; the database 108 is used to store the data required for the operation of the intelligent agent under test. The intelligent agent under test can also be deployed on a cloud server, and the deployment form is not limited in one or more embodiments of this specification.
[0062] In this context, the tested intelligent agent refers to a conversational intelligent agent undergoing clarification capability evaluation. The evaluation system does not access or modify the internal structure and state of the tested intelligent agent, but interacts with it only in a black-box manner: providing input to the tested intelligent agent and receiving actions output by the tested intelligent agent. Therefore, the tested intelligent agent can be a locally deployed self-developed model or a third-party model called through an interface. This specification does not limit this in one or more embodiments.
[0063] The second server cluster 114 is a server cluster on the evaluation system side, including a second database 116 and a second processing engine 118. The second processing engine 118 is used to execute the evaluation methods of one or more embodiments of this specification, including conducting multi-turn dialogues with the tested intelligent agent, generating response text, and calculating evaluation results; the second database 116 is used to store data such as hidden constraint sets, restored constraint sets, dialogue content, and evaluation results.
[0064] Network 112 is a communication network between the first server cluster 106 and the second server cluster 114, such as the Internet or a dedicated network, used to carry the transmission of the first instruction and the interactive data of multi-round dialogue.
[0065] It should be noted that the above division of hardware entities is only an example. The tested intelligent agent and the evaluation system can also be deployed on the same server cluster, and communication via network 112 can be replaced by internal interface calls within the device.
[0066] like Figure 1 The evaluation methods of one or more embodiments in this specification are in Figure 1The architecture can be executed as follows: The second processing engine 118 executes step S102 to obtain the first instruction and the set of hidden constraints. There are two ways to obtain these: first, from the terminal device 102, for example, by receiving the first instruction and the set of hidden constraints configured or uploaded by the evaluator through the terminal device 102; second, offline from the database server 104, i.e., reading the evaluation task pre-built and stored in the task library. Then, the second processing engine 118 executes step S104 via network 112 to send the first instruction. The first instruction is sent to the first processing engine 110 (the agent under test) of the first server cluster 106. It is important to emphasize that the data sent only contains the first instruction; the set of hidden constraints is not included in the transmission, thus ensuring the invisibility of the set of hidden constraints to the agent under test. Afterwards, the second processing engine 118 and the first processing engine 110 execute step S106 to conduct a multi-round dialogue. The first processing engine 110 outputs an action (e.g., a questioning action) based on a first instruction and sends it to the second processing engine 118 via network 112. The second processing engine 118 searches for target hidden constraint text matching the question from the hidden constraint set according to predefined rules, generates response text, and returns it to the first processing engine 110 via network 112. This interaction process is repeated multiple times until the tested agent completes the target operation or meets a preset termination condition. The dialogue content of the multiple rounds can be stored in the second database 116. Finally, the second processing engine 118 (evaluation system) executes step S108 to evaluate the ability based on the dialogue content. After the multiple rounds of dialogue, the second processing engine 118 calculates evaluation metrics based on the dialogue content (e.g., restoring the constraint set, task execution results) to obtain the evaluation result of the tested agent's clarification ability. The evaluation result can be stored in the second database 116 and output.
[0067] In the above architecture, the actions of the evaluation system in steps S102, S104, and S106, as well as step S108, are all executed by the second processing engine 118 of the second server cluster 114. That is, the main body of the evaluation method in one or more embodiments of this specification is the evaluation system; the tested intelligent agent (first processing engine 110) only participates in multi-round dialogue as a black-box interaction object, and the evaluation system does not access or modify its internal logic.
[0068] The following describes embodiments of the evaluation method for the clarification capability of a conversational agent provided in one or more embodiments of this specification, in order to facilitate understanding of the details of the complete solution.
[0069] Figure 2 A flowchart illustrating a method for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; as shown. Figure 2 As shown, this method can be applied to the server cluster in the above architecture, and the method may include the following steps: Step S202: Obtain a first instruction and a set of hidden constraints, wherein the first instruction is used to instruct the tested agent to perform a target operation and does not contain at least one hidden constraint text required to perform the target operation, the set of hidden constraints includes at least one hidden constraint text, and the set of hidden constraints is not visible to the tested agent.
[0070] It should be noted that the target operation refers to the action that the tested intelligent agent needs to perform on behalf of the user, such as purchasing goods that meet the user's needs, searching for and recommending job positions that meet the user's needs, or handling transactions that meet the user's conditions. The quality of the target operation depends on whether the operation result meets the user's constraints.
[0071] The first instruction is the task instruction that directs the aforementioned target operation. It only expresses a summary of the user's needs and does not include all the constraints required to perform the target operation. For example, "Buy me a shirt" indicates the target operation of buying a shirt, but does not include constraints such as color, material, or budget. The first instruction is not necessarily devoid of any constraints; it can contain some constraints, as long as at least one constraint is not included. The number of constraints included in the first instruction can be determined by the difficulty level in the evaluation process. The higher the difficulty of the evaluation, the fewer constraints are included, meaning fewer constraints are initially determined by the agent.
[0072] Hidden constraint text refers to the textual expression of a user's actual needs and preferences that are not stated in the first instruction, such as "blue" or "under 200 yuan". The hidden constraint set is a collection of at least one hidden constraint text that matches the target operation indicated by the first instruction; that is, all hidden constraint texts in the set represent constraints required to complete the target operation. Each hidden constraint text can belong to at least one attribute category in a predefined attribute category set, such as budget, color, or material. The hidden constraint set is stored only on the evaluation system's side and is not sent to the tested agent. The tested agent cannot directly obtain the contents of the hidden constraint set; it can only indirectly obtain them by asking the evaluation system questions and receiving responses.
[0073] That is, by constructing an information gap between the tested intelligent agent and the real user needs through the task structure of the first instruction and the hidden constraint set, the tested intelligent agent can only know the task goal and some constraints, but cannot know the complete constraints.
[0074] In an alternative implementation, the first instruction and the set of hidden constraints are pre-constructed by processing the original instruction. Figure 3 A schematic diagram illustrating the construction of an evaluation task for a method of evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; as shown. Figure 3As shown, the original instruction is a complete task instruction containing all user needs and preferences, such as "Buy me a blue cotton shirt, budget under 200 yuan". During construction, according to predefined attribute slots, at least one constraint corresponding to an attribute in the original instruction is removed. The removed constraints form a hidden constraint set as hidden constraint text, and the remaining instruction text after removal is the first instruction. Figure 3 As shown, the constraints [blue, pure cotton, under 200 yuan] are removed from the original instruction to form a hidden constraint set. The remaining task instruction, "Buy me a shirt," is used as the first instruction. The first instruction can be sent to the tested agent, while the hidden constraint set serves as data used internally by the evaluation system. As mentioned earlier, the number of removed attributes can be related to the difficulty level of the task; the more attributes removed, the higher the task difficulty.
[0075] In addition, the first set of instructions and hidden constraint sets can also be obtained from the historical instruction data of real users, or written by annotators according to predefined attribute categories. This specification does not limit the source of task data in one or more embodiments.
[0076] In an optional implementation, each hidden constraint text in the hidden constraint set belongs to at least one attribute category in a predefined attribute category set. The attribute category set is system-level pre-configured data. For example, for shopping-related operations, the attribute category set may include several categories such as budget, size, color, material, and brand; for employment consultation-related operations, the attribute category set may include several categories such as salary, place of employment, and industry.
[0077] In an optional implementation, the evaluation system can store multiple pre-constructed evaluation tasks in the form of a task library. For example, it can store information such as the first instruction, hidden constraint set, original instruction, difficulty level, and data source identifier of each task in JSONL format. When performing the evaluation, the evaluation system reads an evaluation task from the task library, thus completing the acquisition operation.
[0078] It should be noted that the above process of constructing the first instruction and the set of hidden constraints mentions the difficulty level. The following is a detailed explanation of how the difficulty level determines the set of hidden constraints.
[0079] The scope of the constraint set is determined by the target operation indicated by the first instruction. That is, each constraint text in the constraint set is a constraint condition that must be satisfied to complete the target operation and is related to the object of the target operation. Preference information unrelated to the object of the target operation is not included in the constraint set. For a constraint set where the target operation is determined, it can be split into the information carried by the first instruction and the hidden constraint text in the hidden constraint set. The basis for this splitting can be the difficulty level.
[0080] The difficulty level can be configured within the evaluation task to indicate the level of clarification required for the tested agent to fill in the information gaps. When removing attributes from the original instructions to construct the evaluation task, the number of attributes removed is positively correlated with the difficulty level: for low-difficulty tasks, fewer attributes are removed, such as 1 or 2. Correspondingly, the number of hidden constraint texts in the hidden constraint set is smaller, and more constraints are retained in the first instruction, requiring only a few clarification questions for the tested agent to fill in the information gaps. For high-difficulty tasks, more attributes are removed, such as more than 3. Correspondingly, the number of hidden constraint texts in the hidden constraint set is larger, and fewer constraints are retained in the first instruction, or even only the task objective is retained, requiring multiple rounds of targeted clarification questions for the tested agent to fill in the information gaps.
[0081] Furthermore, the difficulty level can also be related to the importance of the removed attributes. For example, weights can be pre-configured for each attribute category, and these weights represent the degree to which the constraints of that attribute category affect the quality of the target operation. For tasks with high difficulty levels, attributes with higher weights are removed first, so that if the tested agent does not clarify such constraints, its task execution result will be more affected, thus distinguishing the merits of the clarification strategy.
[0082] As mentioned earlier, the difficulty level can be stored synchronously with the first instruction and hidden constraint set as metadata of the evaluation task, for example, recorded in the tier field of the corresponding task in the task library. In the subsequent evaluation stage, the evaluation system can perform hierarchical statistics on the evaluation results according to the difficulty level, thereby analyzing the trend of the clarification ability of the tested agent changing with the task difficulty, and avoiding the distortion of evaluation conclusions caused by mixed statistics of tasks of different difficulties.
[0083] Step S204: Send a first instruction to the agent under test, and conduct multi-round dialogue with the agent under test based on the first instruction and the set of hidden constraints.
[0084] It should be noted that multi-turn dialogue refers to alternating, multi-round interactions between the evaluation system and the tested agent. Each round of interaction includes: the tested agent outputting an action, and the evaluation system returning a response to the tested agent in response to that action. The actions output by the tested agent include at least questioning actions, and may also include execution actions such as searching, browsing, purchasing, and recommending, the specific actions being determined by the agent's main functions; the response returned by the evaluation system includes the reply text to the question, and the execution result of the execution action. That is, multi-turn dialogue is not limited to question-and-answer rounds; rounds of execution actions are also part of multi-turn dialogue.
[0085] In an optional implementation, the evaluation system establishes a multi-turn dialogue with the agent under test through an interactive interface. The evaluation system first sends a first instruction as task input to the agent under test, which then makes a decision and outputs an action based on the first instruction. The evaluation system parses the actions output by the agent under test and processes them according to the action type: If the agent under test asks a question, the evaluation system searches for a target hidden constraint text that matches the question from the set of hidden constraints according to predefined rules. If a match is found, a response text is generated based on the target hidden constraint text and sent to the agent under test. If no match is found, a response text is generated based on the question type according to a pre-configured strategy and sent to the agent under test. The content of the response text is limited to the set of hidden constraints to ensure that the response does not reveal information beyond the preset preferences. The specific process of generating the response text will be explained below and will not be repeated here.
[0086] If the tested intelligent agent outputs an action such as searching or browsing, the evaluation system delegates the action to the execution environment for execution and sends the execution result returned by the execution environment as a response to the tested intelligent agent. The execution environment can be a simulated transaction environment or a real activity environment, such as directly handing it over to a shopping platform to perform the operation of searching for products and returning the information of the searched products as a response to the tested intelligent agent. This specification does not limit this in one or more embodiments.
[0087] In an optional implementation, the multi-turn dialogue terminates when certain conditions are met. These conditions include: the tested agent completing the target operation; the number of turns in the multi-turn dialogue reaching a preset maximum; or a preset abnormal situation occurring during the interaction. If the tested agent completes the target operation, the evaluation system can also perform constraint satisfaction verification on the result of the target operation to determine the task execution outcome.
[0088] In an optional implementation, the evaluation system records the entire dialogue content of multiple rounds of conversation. The recording results may include information such as the initiator of each round of interaction, the action type, the text content, and the execution result, and are persistently stored in a structured data format, such as JSONL. Subsequent evaluation steps calculate metrics based on these recording results, thereby supporting cross-experiment comparisons of evaluation results.
[0089] It should be noted that the behavior of the tested agent in a multi-turn dialogue is generated by its own decisions, and the evaluation system does not interfere with the decision-making logic of the tested agent. Therefore, this step is applicable to the evaluation of the clarification ability of any conversational agent.
[0090] Step S206: Based on the dialogue content of the multi-turn dialogue, obtain the evaluation results of the clarification ability of the tested intelligent agent.
[0091] As mentioned earlier, the dialogue content can be all the interaction data generated during the multi-turn dialogue, including at least the actions and text content output by the tested agent, the response texts generated by the evaluation system, the execution results of execution-type actions, the set of recovered constraints accumulated during the multi-turn dialogue, and the task execution results. The set of recovered constraints is the set of constraint texts recovered during the process, which will be explained in detail later and will not be repeated here.
[0092] Clarification ability is the comprehensive ability of a tested agent to fill in constraints by actively asking questions when faced with information gaps and then applying the acquired constraints to the target operation. Clarification ability includes at least three levels: the ability to ask appropriate clarifying questions, the ability to extract constraint information from responses, and the ability to integrate the acquired constraint information into subsequent execution. The evaluation result refers to the data obtained after quantitatively evaluating the clarification ability.
[0093] It should be noted that the evaluation system can calculate the attribute recovery rate based on the recovery constraint set, that is, the ratio of the number of hidden constraint texts in the recovery constraint set to the total number of hidden constraint texts in the hidden constraint set is determined as the attribute recovery rate. The evaluation system can also further calculate the constraint integration rate: detect whether the hidden constraint texts in the recovery constraint set are included in the search query of the tested agent, and determine the constraint integration rate based on the number of hidden constraint texts integrated into the search query and the number of hidden constraint texts in the recovery constraint set. The evaluation system can also conduct multi-turn dialogues with the tested agent under at least two different behavioral profiles to obtain the task execution results corresponding to each behavioral profile, and determine the profile sensitivity based on the differences between the task execution results. The various dimensions of the evaluation indicators will be described in detail later, and will not be repeated here. In addition, the various dimensions of the evaluation indicators can be arbitrarily combined to form the evaluation results, and one or more embodiments in this specification do not limit the specific dimensions included in the evaluation results.
[0094] By employing the above method, the evaluation system sends only the first instruction carrying the target operation to the tested agent, and makes the hidden constraint set invisible to the tested agent, thus isolating the data between the tested agent and the actual needs. In this way, the tested agent can only complete the constraints by clarifying the questions, thereby recreating the interaction scenario where the user's needs are not fully expressed under controlled conditions. During multi-turn dialogues, the evaluation system's responses are limited to the predefined hidden constraint set, avoiding information leakage caused by simulating users deviating from preset preferences. Furthermore, the responses generated based on predefined rules are deterministic. Under the same task configuration (such as the same hidden constraint set and the same target operation), the interaction process and results of multiple evaluations are consistent, making the evaluation results comparable between different tested agents and between different versions of the same tested agent. This solves the problems of non-reproducible results and incomparable evaluation conclusions in existing evaluation methods.
[0095] In an optional implementation, a multi-turn dialogue is conducted with the agent under test based on the first instruction and the set of hidden constraints, including: for each question initiated by the agent under test, searching for target hidden constraint text that matches the question from the set of hidden constraints according to predefined rules, wherein the set of hidden constraints matches the target operation indicated by the first instruction; if the target hidden constraint text is found, generating a response text for the question based on the target hidden constraint text and sending it to the agent under test; if the target hidden constraint text is not found, generating a response text for the question based on the question type according to a pre-configured strategy and sending it to the agent under test.
[0096] It should be noted that target hidden constraint text refers to the hidden constraint text in the hidden constraint set that matches the current question and is selected for generating the response text. A question may correspond to one or more target hidden constraint texts, or there may be no matching target hidden constraint text. Matching refers to a content-based correspondence between the question and the hidden constraint text. For example, the question text may contain a certain hidden constraint text, or the attribute category involved in the question may be consistent with the attribute category to which a certain hidden constraint text belongs, and the content relevance meets preset conditions.
[0097] Question type refers to the classification of questions based on their content characteristics, such as irrelevant questions, open-ended questions, categorized questions, and broad questions. Question type is used to determine the response generation strategy when the target hidden constraint text is not found. Pre-configured strategies are pre-set response generation methods for each question type; for example, generating generalized or negative feedback responses for irrelevant questions, and generating confusing or deflective responses for categorized questions.
[0098] It should be noted that questions come in various forms. Using a single method to find matching constraints cannot account for the characteristics of different question types. Therefore, it is possible to improve the accuracy of the search by first classifying the questions and then selecting the search method based on the type. In an optional implementation, the process of searching for target hidden constraint text that matches the question from the hidden constraint set according to predefined rules includes: classifying the questions to obtain the question types; and searching for target hidden constraint text that matches the question from the hidden constraint set according to the search method that matches the question type. It should be noted that question type refers to the classification of questions based on their content characteristics. As mentioned earlier, question types can include irrelevant questions, open questions, category questions, and broad questions: irrelevant questions are topic-related questions unrelated to the target operation; open questions are general inquiries about requirements without targeting a specific attribute category; category questions are questions targeting a predefined attribute category; and broad questions cover multiple attribute categories simultaneously.
[0099] Furthermore, real users are not always completely open to answering questions, and their level of cooperation varies from person to person. Behavioral profiling can be used to limit the number of constraints revealed in a single response, allowing simulated user responses to exhibit controllable diversity. In an optional implementation, generating response text based on the target hidden constraint text includes: generating response text based on a pre-configured response strategy corresponding to the behavioral profile and the target hidden constraint text, wherein the behavioral profile is used to limit the number of hidden constraint texts allowed to be revealed in the response text.
[0100] Behavioral profiling refers to the parameterized configuration of simulated user behavior patterns. It limits the number of hidden constraint texts that can be revealed in the response text, and the response method when a match fails. Behavioral profiling allows the same evaluation system to simulate users with different levels of cooperation, such as ordinary users, cautious users, and cooperative users. Revealing refers to the behavior of informing the tested agent of the content of hidden constraint texts in the response text; unrevealed hidden constraint texts remain invisible to the tested agent.
[0101] In an optional implementation, after receiving a question, the evaluation system first categorizes the question. After categorization, the system searches for the target hidden constraint text using a search method that matches the question type. For example, for categorized questions, the system can first perform string matching between the question text and each hidden constraint text in the hidden constraint set; if a match is found, the target hidden constraint text is directly determined. For open questions, the search method involves selecting from the entire hidden constraint set according to a preset sorting, with the number of selections limited by the behavioral profile.
[0102] In an optional implementation, the behavioral profile is determined by a combination of configuration parameters, such as: parameters limiting the maximum number of open questions that can be revealed in a single response, parameters indicating whether open questions should be revealed according to the maximum number, parameters indicating the response method when a match fails, and parameters indicating the response method for irrelevant questions. By adjusting the combination of configuration parameter values, new behavioral profiles can be expanded without modifying the program logic.
[0103] Through one or more embodiments of this specification, by adopting a classification-then-search processing method, the search method is adapted to the semantics of the question, avoiding mismatches of different types of questions by a single search logic, and improving the accuracy of target hidden constraint text selection and the relevance of the response content to the question content.
[0104] The specific classification method for question types is explained below: In an optional implementation, questions are classified to obtain question types, including: if the matching degree between the question text and the irrelevant text set exceeds a first matching degree threshold, the question is determined to be an irrelevant question, wherein the irrelevant text set is used to represent topics unrelated to the target operation; if the matching degree between the question text and the open text set exceeds a second matching degree threshold, the question is determined to be an open question, wherein the open text set includes a list of open question keywords; if the question text matches at least one attribute category in the attribute category set, the question is determined to be a category question, wherein each hidden constraint text in the hidden constraint set belongs to at least one attribute category in the attribute category set; if the number of attribute categories matched by the category question exceeds a preset number threshold, or if the question contains a conjunction, the category question is determined to be a broad question, wherein the conjunction is used to connect multiple attribute types in parallel.
[0105] It's important to note that the irrelevant text set refers to a predefined collection of texts representing topics unrelated to the target operation, such as keywords or sentence templates for topics like weather, greetings, and emotions. The specific content of the irrelevant text set can be configured according to the domain of the target operation. The open text set refers to a predefined collection of texts representing sentences that broadly inquire about needs, such as a list of keywords for open-ended questions like needs, requirements, preferences, and what one wants. It can also include regular expression patterns that match broad question sentences. The essence of an open question is that it doesn't point to any specific attribute category.
[0106] The attribute category set refers to a collection of predefined attribute categories at the system level, such as budget, size, color, material, and brand. Each attribute category corresponds to a category keyword. The hidden constraint text in the hidden constraint set belongs to at least one attribute category within that set. This attribution relationship is a static data relationship pre-configured before evaluation. For example, color is an attribute category, and blue is its corresponding hidden constraint text.
[0107] Broad questions are a special type of category question that overloads the scope of the question; they superficially point to an attribute category, but cover too many attribute categories, making it difficult for users to fully answer in a single response. Connectives are words used to link multiple attribute category values in parallel, such as "or" or "or." A question containing connectives that link values from multiple attribute categories indicates that the question attempts to obtain constraint information from multiple categories in a single query.
[0108] In an optional implementation, the evaluation system categorizes questions according to a preset detection order. Specifically, it first matches the question text with a set of irrelevant texts, calculates the matching degree between the two, and determines the question as an irrelevant question if the matching degree exceeds a first matching degree threshold, thus ending the classification; for example, the question "How's the weather today?" has a keyword matching degree with the weather topic that exceeds the threshold, and is therefore determined to be an irrelevant question.
[0109] If the question is not determined to be irrelevant, the question text is matched against the open text set. The open text set can include a list of open question keywords and regular expression patterns. If the match score exceeds a second match score threshold and the question does not hit any attribute category in the attribute category set, the question is determined to be an open question; for example, the question "What are your requirements?" hits the keyword "requirements" but does not point to any attribute category, and is therefore determined to be an open question. It should be noted that setting the condition of not hitting any attribute category avoids misidentifying questions like "What are your requirements for colors?" which essentially point to the color category, as open questions.
[0110] A question is classified as a category question if it matches at least one attribute category in the attribute category set. Attribute category matching is achieved through category keywords; for example, a question containing "color" matches the "color" category. Each hidden constraint text in the hidden constraint set is pre-assigned to at least one attribute category in the attribute category set. Therefore, once the matching attribute category is determined, subsequent searches can be performed within the constraints belonging to that category.
[0111] The evaluation system can also perform further broad detection on category questions. If the number of attribute categories matched by a category question exceeds a preset threshold (e.g., 3), or if the question contains conjunctions (e.g., "or" or "or") used to connect multiple attribute category values in parallel, the category question is identified as a broad question. For example, the question "Do you want red, plus size, or silk?" involves values from three categories: color, size, and material, and contains the conjunction "or", and is identified as a broad question.
[0112] Through one or more embodiments of this specification, questions are accurately classified into types that match their semantics, providing a reliable basis for subsequent selection of search methods and response strategies by type, and ensuring a high relevance between the response content and the question content; the entire classification process is completed based on deterministic matching of predefined text sets, without introducing random factors, and each text set and threshold can be configured, which is more conducive to the reproducibility of the evaluation system.
[0113] As mentioned earlier, if the problem type is determined to be a categorical problem, the defined target hidden constraint text can be returned to the tested agent. The following explains how to find the target hidden constraint text from the set of hidden constraints.
[0114] In an optional implementation, when the question type is a category question, searching for target hidden constraint text that matches the question from the hidden constraint set includes: performing string matching between the question text and each hidden constraint text in the hidden constraint set; if the question text contains a string that matches any hidden constraint text, determining the hidden constraint text that matches the question text as the target hidden constraint text; if the question text does not contain a string that matches any hidden constraint text, determining the attribute category that matches the question text as the target attribute category, and filtering candidate hidden constraint texts belonging to the target attribute category from the hidden constraint set; performing word segmentation on the question text and at least one candidate hidden constraint text respectively, and calculating the overlap score between the word segmentation results; and determining the candidate hidden constraint text whose overlap score meets the preset scoring conditions as the target hidden constraint text.
[0115] It should be noted that the target attribute category refers to the attribute category pointed to by the category question, which can be determined by matching the question text with the category keywords corresponding to each attribute category in the predefined attribute category keyword table. Candidate hidden constraint text refers to the hidden constraint text belonging to the target attribute category in the hidden constraint set.
[0116] The overlap score is a quantitative value that measures the degree of overlap between the segmented results of the query text and the segmented results of the candidate hidden constraint text. For example, it can be expressed as the number of identical word units in the two segmented results, or the ratio of the number of identical word units to the total number of word units in the candidate hidden constraint text. The overlap score is used to measure the relevance between the candidate hidden constraint text and the query content.
[0117] In an optional implementation, after determining that the question is a category question, the evaluation system first performs a first-level search, matching the question text against each hidden constraint text in the hidden constraint set. If the question text contains a string that matches any hidden constraint text, then that hidden constraint text is identified as the target hidden constraint text, and the search ends. For example, if the hidden constraint text is "blue," and the question is "Do you want blue?", the question text contains "blue," so it's a direct match.
[0118] If the first-level search fails, the evaluation system performs a second-level search. First, the target attribute category is determined. The question text is matched against the category keywords corresponding to each attribute category in the predefined attribute category keyword table. The attribute category corresponding to the matched category keyword is then determined as the target attribute category. For example, asking "What are your color preferences?" matches the keyword for the color category, so the target attribute category is the color category.
[0119] Then, the evaluation system uses the pre-configured attribution relationship between hidden constraint text and attribute categories to filter candidate hidden constraint text belonging to the target attribute category from the hidden constraint set. For example, if the hidden constraint set is {blue, under 200 yuan, pure cotton}, and the target attribute category is color, the candidate hidden constraint text would be blue. Next, the evaluation system performs word segmentation on the question text and the candidate hidden constraint text respectively, and calculates the overlap score between the word segmentation results.
[0120] Finally, the evaluation system identifies candidate hidden constraint texts whose overlap scores meet preset scoring criteria as target hidden constraint texts. Preset scoring criteria include, for example, an overlap score greater than or equal to a preset scoring threshold; when multiple candidate hidden constraint texts meet the preset scoring criteria, the one with the highest overlap score is selected. If no candidate hidden constraint text meets the preset scoring criteria, the target hidden constraint text was not found in this search, and a response text is generated according to the question type and a pre-configured strategy.
[0121] It should be noted that the execution order of the first and second level searches described above is only an example. In scenarios where the number of candidate hidden constraint texts is small, some processing steps may be omitted. This specification does not limit this in one or more embodiments.
[0122] Through one or more embodiments of this specification, questions that directly contain constraint values can be quickly found at the first level. Questions that are not directly found are narrowed down by attribute category attribution, and candidate constraints are filtered for relevance by overlap scoring, which ensures a high relevance between the target hidden constraint text and the question content, thereby ensuring the accuracy of the response content.
[0123] As mentioned earlier, the process of generating response text is related to the configured behavioral profile. By setting different behavioral profiles, different user personalities can be imitated. The following section explains the specific process of generating behavioral profiles and response text: In an optional implementation, when the behavior profile is the first profile, the response text is generated based on the pre-configured response strategy and target hiding constraint text corresponding to the behavior profile, including: (1) When the problem type is an open problem, if the number of hidden constraint texts in the hidden constraint set is not greater than the first preset number, the response text is generated based on all hidden constraint texts; if the number of hidden constraint texts is greater than the first preset number, the first N hidden constraint texts are selected according to the preset sorting to generate the response text, and a first prompt message is added to the response text, where N is an integer not less than 1, and the first prompt message is used to instruct the tested agent to recommend and determine the requirements corresponding to the unrevealed hidden constraint texts. (2) When the question type is a category question, generate the response text based on the target hidden constraint text and the matching attribute category.
[0124] It should be noted that the first profile refers to the behavioral profile of a user simulating a normal level of cooperation. Its response characteristics are characterized by moderate disclosure and some reservation; when faced with open-ended questions, only some constraints are revealed, and a flexible approach is taken towards the unrevealed parts. The first profile closely resembles the response behavior of most users in real-world scenarios and is used as the default profile configuration for evaluation in one implementation. The first preset quantity refers to the limit for determining whether to limit the disclosure, for example, 2. When the number of hidden constraint texts in the hidden constraint set is not greater than the first preset quantity, it indicates that the total number of constraints is small and can be fully disclosed; otherwise, limited disclosure is required.
[0125] The preset sort is a pre-determined arrangement of the hidden constraint texts in the hidden constraint set. This can be the storage order of the constraint texts within the set, or an order determined by the importance of the constraint texts. The preset sort is deterministic, ensuring consistent selection results under the same task configuration.
[0126] The first prompt message refers to the suggestive wording attached to the response text during limited disclosure, used to instruct the tested agent to recommend and determine the requirements corresponding to the undisclosed hidden constraint text, such as "Other details can be flexible" or "The rest is not so certain, you can recommend."
[0127] When faced with a general, open-ended question like "What are your requirements?", the simulated user in the first profile typically only states the conditions they care about most, rather than giving all constraints at once. They also remain open to suggestions regarding unmentioned conditions. Through this first-profile response strategy, the tested agent faces an information acquisition environment closer to real-world interaction: it cannot expect the user to reveal everything, yet it can obtain authorization to recommend remaining needs. This allows us to examine the tested agent's clarification and recommendation decision-making abilities when partial information is available.
[0128] In an optional implementation, when the behavioral profile is configured as the first profile, the evaluation system generates response text according to the question type.
[0129] When the problem type is open-ended, the evaluation system first determines the relationship between the number of hidden constraint texts in the hidden constraint set and a first preset number. For example, if the first preset number is 2, and the number of hidden constraint texts is no greater than 2, it indicates that the user preference is low, and the evaluation system generates response text based on all hidden constraint texts, i.e., revealing all constraints at once. If the number of hidden constraint texts is greater than 2, the evaluation system selects the top N hidden constraint texts according to a preset order to generate response text. For example, if N is 2, only the two constraints at the top of the order are revealed, and a first prompt message is added to the response text. The first prompt message may be, for example, "Other details can be flexible" or "The rest are uncertain, you can recommend," meaning that the requirements corresponding to the unrevealed constraints are left to the tested agent to recommend and determine.
[0130] Optionally, the preset sorting can be determined according to the importance of the hidden constraint text. For example, weights can be pre-configured for each attribute category, and the hidden constraint text belonging to the attribute category with higher weights can be sorted first, so that the first profile will reveal the constraints that users care about most and be closer to the behavior habits of real users.
[0131] When the question type is a category question, the evaluation system generates a response text based on the found target hidden constraint text and the attribute category that matches the question. The response text can be generated according to a predefined response template, such as filling the response template with the target hidden constraint text. For example, if the target hidden constraint text is "blue" and the attribute category that matches the question is color, the generated response text could be "I prefer blue". Since the category question itself points to a specific category, and the user's preference under that category is clear, the first profile returns the corresponding constraint truthfully according to the matching results for category questions, without any limit processing.
[0132] In an optional implementation, when the behavior profile is a second profile, response text is generated based on the pre-configured response strategy and target hiding constraint text corresponding to the behavior profile, including: (1) When the question type is an open question, select a hidden constraint text according to the preset sorting to generate a response text, and add a second prompt information to the response text. The second prompt information is used to guide the tested agent to ask questions separately for other attribute categories. (2) When the question type is a category question, generate the response text based on the target hidden constraint text and the matching attribute category.
[0133] It should be noted that the second profile simulates the behavior of a cautious user. When faced with general open-ended questions, this type of user is only willing to reveal one preference and hopes that the other party will inquire about other specific conditions one by one, rather than providing all constraints at once. The level of cooperation of users corresponding to the second profile is lower than that of the first profile mentioned above.
[0134] The second prompt message refers to the guiding wording attached to the response text, used to guide the tested agent to ask questions about other attribute categories separately, such as "Please ask me about other specific attributes separately." The second prompt message does not contain any hidden constraint text; it only reflects the user's response habits.
[0135] The second profile simulates users who don't reveal their preferences unless specifically asked, and only one preference is revealed at a time. If the tested agent only initiates general open-ended questions, it will not be able to efficiently obtain complete constraints. It must fill in the information gaps through multiple rounds of targeted categorized questions. By using the response strategy of the second profile, we can examine the tested agent's ability to adjust its clarification strategy in scenarios with low-cooperation users, that is, whether it can ask specific questions one by one, and whether it can withstand the interaction cost of multiple rounds of follow-up questions.
[0136] In an optional implementation, when the behavioral profile is configured as the second profile, the evaluation system generates response text according to the question type.
[0137] When the question type is open-ended, the evaluation system selects one hidden constraint text from the hidden constraint set according to a preset order to generate the response text. This means that regardless of the number of constraints in the hidden constraint set, a single response reveals at most one constraint. Simultaneously, the evaluation system adds a second prompt to the response text, guiding the tested agent to ask questions about other attribute categories separately. For example, if the hidden constraint set is {blue, under 200 yuan, pure cotton}, and the tested agent asks the open-ended question "What are your requirements?", the evaluation system could generate the response text as "I prefer blue. For other preferences, please ask me about specific attributes separately." Thus, the tested agent can only obtain one constraint at a time. If it needs to obtain other constraints, it must ask category-specific questions for attributes such as budget and material.
[0138] When the question type is a category question, the evaluation system generates a response text based on the found target hidden constraint text and the attribute category that matches the question. For example, if the tested agent initiates a category question "What is your approximate budget?" after being guided by the second prompt information, and the evaluation system determines that the target hidden constraint text is within 200 yuan, the generated response text could be "Budget controlled within 200 yuan".
[0139] Optionally, if the question posed by the tested agent does not find the target hidden constraint text, such as the attribute category pointed to by a category question not having a corresponding constraint in the hidden constraint set, the evaluation system generates a confused response text to indicate that it cannot understand the question, such as "I'm not sure what you're asking".
[0140] In an optional implementation, when the behavior profile is a third profile, the response text is generated based on the pre-configured response strategy and target hiding constraint text corresponding to the behavior profile, including: (1) When the question type is an open question, generate the response text based on all hidden constraint texts in the hidden constraint set, and do not attach any hint information to the response text; (2) When the question type is a category question, generate the response text based on the target hidden constraint text and the matching attribute category.
[0141] It should be noted that the third profile simulates the behavior of a highly cooperative user. When faced with an open-ended question, it reveals all constraints at once without attaching any limiting conditions. The user corresponding to the third profile demonstrates a higher level of cooperation than the first and second profiles, representing idealized information acquisition conditions. Furthermore, it does not include any additional prompts; the response text does not contain any of the first type of prompt (recommendation / authorization prompts) or the second type of prompt (guided question prompts). The response text consists solely of the content corresponding to the hidden constraint text and does not include any expression of the user's attitude towards the remaining requirements.
[0142] The third profile simulates users with ideal cooperation levels. Under the third profile, the tested agent can obtain all hidden constraint texts through a single open question, and there are almost no obstacles in the information acquisition process. If the task execution result of the tested agent still does not meet the constraint conditions, then the problem is not in the clarification process, but in the execution process such as search and decision-making.
[0143] In an optional implementation, when the behavioral profile is configured as the third profile, the evaluation system generates response text according to the question type.
[0144] When the question type is open-ended, the evaluation system generates a response text based on all hidden constraint texts in the hidden constraint set. This means there is no quantity limit; all constraints are revealed at once, and no additional prompts are added to the response text. For example, if the hidden constraint set is {blue, under 200 yuan, pure cotton}, and the tested agent asks the open-ended question "What are your requirements?", the evaluation system could generate a response text like "I want a blue, pure cotton shirt, with a budget under 200 yuan." The response text is generated, for example, by concatenating or filling in all hidden constraint texts according to a predefined template. Since all constraints have been revealed, there are no unrevealed remaining requirements, so no additional recommendation or authorization prompts are needed in the response text; similarly, due to the user's high level of cooperation, no additional guiding questions are required.
[0145] When the question type is a category question, the evaluation system generates a response text based on the found target hidden constraint text and the attribute category that matches the question. The processing method is the same as that for category questions in the first and second profiles mentioned above, that is, the corresponding constraint is returned truthfully according to the matching result. For example, if the tested agent raises a category question "What material do you want?", and the evaluation system determines that the target hidden constraint text is "pure cotton", the generated response text can be "Pure cotton material is better".
[0146] Optionally, the third profile can be used in conjunction with the first and second profiles for profile sensitivity evaluation: the behavioral profiles are configured as the first, second, and third profiles, respectively, and multiple rounds of dialogue are conducted with the tested agent under the same evaluation task. The differences in task execution results under the three profile configurations are compared. If the task execution result of the tested agent under the third profile is significantly better than that under the second profile, it indicates that its clarification ability is sensitive to user cooperation; if the task execution results under the three profiles are similar, it indicates that its clarification ability has good robustness.
[0147] Through one or more embodiments of this specification, the first profile, the second profile, and the third profile form a complete cooperation gradient in terms of the number of disclosures: limited, single, and all. The three profiles are used together to conduct profile sensitivity evaluation, which can quantify the difference in the test agent's clarification ability to the change in user cooperation and further locate whether the test agent's ability problem is in the clarification stage or the execution stage.
[0148] In explaining the three behavioral profiles above, we focused on the response text generation when there are hidden constraint texts for the target. The following explains the situation where the question cannot be matched with constraints from the set of hidden constraints.
[0149] It should be noted that the evaluation system cannot fabricate constraint information, nor can it fail to respond. Therefore, it is necessary to configure response methods that conform to the behavioral profile for different situations. In an optional implementation, when no target hidden constraint text is found, a response text is generated based on the question type according to a pre-configured strategy, including: when the question type is an irrelevant question and the behavioral profile is a first profile or a third profile, a generalized response text is generated according to a pre-configured irrelevant question handling strategy, indicating that there is no additional need for the target operation; when the question type is an irrelevant question and the behavioral profile is a second profile, a negative feedback text is generated according to a pre-configured irrelevant question handling strategy, indicating negative feedback to the question; when the question type is a category question and the behavioral profile is a second profile, a confused response text is generated, indicating that the question cannot be understood; when the question type is a category question and the behavioral profile is a first profile or a third profile, a deflection response text is generated, indicating that there is no preference for the attribute category involved in the question.
[0150] In an optional implementation, when the evaluation system fails to find the target hidden constraint text, it generates response text based on the question type and the currently configured behavioral profile.
[0151] When the question type is irrelevant, the evaluation system generates response text according to a pre-configured irrelevant question handling strategy. This strategy is associated with the behavioral profile. When the behavioral profile is the first or third profile, a deflection strategy is used to generate a generalized response text. For example, if the tested agent asks "How's the weather today?", the evaluation system generates a generalized response text like "It's fine," indicating no additional need for the target operation and prompting the dialogue to return to the target operation. When the behavioral profile is the second profile, a penalty strategy is used to generate negative feedback text. For example, if the tested agent asks the same irrelevant question, the evaluation system generates negative feedback text like "How does this relate to my needs?", expressing negative feedback on the question. Negative feedback forms a clear negative signal against the tested agent's off-topic behavior. Whether the tested agent can correct the question's direction after receiving negative feedback can also be used as an evaluation dimension of clarification ability.
[0152] When the question type is a categorical question, but the hidden constraint set does not contain a corresponding constraint for that category, the evaluation system still differentiates responses based on behavioral profiles: If the behavioral profile is the second profile, a confused response text is generated, such as "I'm not quite sure what you're asking," simulating the behavior of a cautious user who refuses to give a direct answer to an ambiguous question; if the behavioral profile is the first or third profile, a deflected response text is generated, such as "I don't have any specific requirements in this regard," indicating no preference for the attribute category involved in the question. The deflected response text only expresses no preference for the category involved in the question and does not reveal constraint information for other categories. If the tested agent needs to obtain other constraints, it must continue to ask questions about other attribute categories.
[0153] It should be noted that the above four types of response texts are all predefined wording formats. The evaluation system selects and generates them according to the situation. The wording does not contain any constraints from the hidden constraint set, nor does it fabricate new preference information on the spot.
[0154] In an optional implementation, the parameterized configurations of the first image, the second image, and the third image can be as shown in Table 1 below: Table 1 Image Configuration Parameters
[0155] As shown in Table 1, the upper limit of the number of open questions revealed (the corresponding parameter is open_top_k) limits the maximum number of hidden constraint texts that the tested agent can reveal in a single response text when initiating an open question. The first profile is set to 2: ordinary users only state the two preferences they care about most when faced with a general question; the second profile is set to 1: cautious users only reveal one preference at a time; the third profile is set to None (all), indicating that there is no upper limit.
[0156] The open question limit enable / disable switch (corresponding parameter is open_use_top_k_rule) is the master switch for the upper limit of the number of open questions that can be revealed. When set to True (enabled), the responses to open questions are truncated to the limit; when set to False (disabled), the upper limit is ignored and all open questions are revealed. The Third Portrait achieves full disclosure by setting this parameter to disabled.
[0157] The strict mode switch (corresponding parameter is strict_mode) limits the response method when the target hidden constraint text is not found: True (on) generates a confusing response text (I'm not quite sure what you're asking), and False (off) generates a deflected response text (I don't have specific requirements for this). The second profile is turned on, corresponding to the confused reaction of cautious users to ambiguous questions; the first and third profiles are turned off, corresponding to the easygoing response of normal and cooperative users.
[0158] The irrelevant question handling strategy (corresponding parameter is `irrelevant_policy`) limits the response method when the tested agent raises a question unrelated to the target operation: when `deflect` (generalized response) is selected, a generalized response text is generated (either is acceptable); when `penalize` (negative feedback response) is selected, a negative feedback text is generated. The second profile uses a negative feedback response—cautious users have low tolerance for off-topic questions, so negative feedback is given; the first and third profiles use generalized responses to guide the dialogue back to the target operation without requiring further explanation.
[0159] As can be clearly seen from the table above, the setting of the behavior profile is simply different combinations of parameters. Each of the three sets of values constitutes a user's personality. This specification does not limit the number of behavior profiles that can be set in one or more embodiments. Users can set the parameters according to their needs.
[0160] The foregoing section provided a detailed explanation of the generation of response text in multi-turn dialogues. If one or more embodiments of this specification are considered as a complete evaluation system, all the methods described above for generating response text can be seen as a part of the system, or a module specifically designed for text-based dialogue with the agent under test. This involves a user simulator that binds response behavior to a predefined set of hidden constraints. Figure 4A schematic diagram of the architecture of a user simulator for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; as shown. Figure 4 As shown, for ease of understanding, it is assumed that the tested intelligent agent and the evaluation system run on two different server clusters. Figure 4 In this system, the evaluation system may include a user simulator and an evaluation pipeline. The user simulator is used to simulate the dialogue ability of real users and engage in multi-turn dialogue interactions with the agent under test, while the evaluation pipeline is responsible for evaluating based on the dialogue content.
[0161] Specifically, the user simulator is a logical functional module in the evaluation system used to simulate multi-turn dialogues between real users and the tested intelligent agent. Its input is a question initiated by the tested intelligent agent, and its output is the response text to that question. The user simulator is a user simulation system with constrained response behavior. Its response content is limited to a pre-configured set of hidden constraints, and its response method is driven by predefined rules and behavioral profiles, thereby ensuring the determinism and controllability of response behavior while faithfully simulating user response behavior.
[0162] It should be noted that the user simulator can be an independent functional module in the evaluation system, or it can be a collection of logical actions executed by the controller of the evaluation system. This specification does not limit its specific implementation form in one or more embodiments.
[0163] Figure 5 A schematic diagram of a user simulator for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; as shown. Figure 5 The image shows the internal logic architecture of the user simulator.
[0164] First, let's explain the data that the user simulator depends on. It should be noted that detailed explanations of the data used can be found above; this section only provides a brief summary and explanation.
[0165] The data includes: (1) a set of hidden constraints, i.e., user needs and preferences that actually exist but are not expressed in the first instruction; (2) a set of predefined texts, including a set of irrelevant texts for identifying irrelevant questions, a set of open texts for identifying open questions (e.g., a list of open question keywords), and a list of attribute category keywords for establishing a mapping relationship between questions and attribute categories; and (3) a behavioral profile configuration, i.e., a parameterized configuration that simulates user behavior patterns, used to limit the number of hidden constraint texts that can be revealed in a single response text, the response method when a match fails, and the handling method for irrelevant questions. Behavioral profiles include, for example, a first profile simulating a normal user, a second profile simulating a cautious user, and a third profile simulating a highly cooperative user.
[0166] Then, its internal logic will be explained. Similarly, the specific internal logic can be found in the text above. Here, we will only provide a simple summary and explanation of the logic.
[0167] The logic includes: (1) Question classification: After the user simulator receives a question sent by the tested agent, it first classifies the question to obtain the question type. The classification is based on the matching of a predefined text set. If the matching degree with the irrelevant text set exceeds the first matching degree threshold, it is determined to be an irrelevant question; if the matching degree with the open text set exceeds the second matching degree threshold and does not match any attribute category, it is determined to be an open question; if it matches at least one attribute category in the attribute category set, it is determined to be a category question; if the number of attribute categories matched by the category question exceeds the preset number threshold or the question contains a conjunction that connects multiple attribute categories in parallel, it is further determined to be a broad question.
[0168] (2) Attribute Category Matching: The user simulator searches for target hidden constraint text that matches the question in the hidden constraint set according to the search method that matches the question type. For category questions, the search is divided into two levels. First, the question text is matched with each hidden constraint text. If a match is found, the target hidden constraint text is directly determined. If no match is found, the target attribute category corresponding to the question is determined. Candidate hidden constraint texts belonging to this category are filtered from the hidden constraint set. After word segmentation and calculation of overlap score, candidates whose scores meet the preset scoring conditions are determined as target hidden constraint texts.
[0169] (3) Response Generation: When the target hidden constraint text is found, a response can be generated directly for category questions. For other cases, the user simulator generates response text based on the response strategy corresponding to the pre-configured behavioral profile and the target hidden constraint text. The behavioral profile limits the number of constraints that can be revealed in a single response: Under the first profile, all constraints are revealed if the number of constraints is not greater than the first preset number; otherwise, the top N constraints are selected according to the preset sorting and a first prompt message is attached to indicate that the tested agent should recommend and determine the unrevealed needs; Under the second profile, at most one constraint is revealed at a time and a second prompt message is attached to guide the tested agent to ask questions about other attribute categories separately; Under the third profile, all constraints are revealed without any additional prompt message. When the target hidden constraint text is not found, the user simulator generates a response in a predefined wording format according to the question type and behavioral profile: Irrelevant questions generate a generalized response text indicating no supplementary needs under the first and third profiles, and a negative feedback text indicating negative feedback under the second profile; category questions generate a confused response text indicating that the question cannot be understood under the second profile, and a deflection response text indicating no preference for the attribute category under the first and third profiles. None of the above statements contain any content from the hidden constraint set.
[0170] It should be noted that the agent being tested interacting with the user simulator is a black box; the evaluation system cannot directly read its internal state to confirm whether it has remembered the constraints. However, the response text is shared by both parties. As long as the constraint text appears in the response, the agent being tested has encountered this information within the dialogue context. Therefore, constraint recovery detection can be performed on each round of response text, and the recovered constraint set can provide intuitive data support for subsequent evaluations. Thus, a mechanism is needed to record the recovered hidden constraint text. In an optional implementation, after generating the response text to the question, the method further includes: performing string matching between the response text and each hidden constraint text in the hidden constraint set; identifying the hidden constraint text contained in the response text as the recovered hidden constraint text, and adding the recovered hidden constraint text to the recovered constraint set. The recovered constraint set is used to accumulate and record the recovered hidden constraint text during multiple rounds of dialogue.
[0171] The recovered hidden constraint text refers to the hidden constraint text that has appeared in the response text generated by the evaluation system. The tested agent, by asking questions, transforms this hidden constraint text from an invisible state to a visible state in the dialogue, meaning the constraint information has been acquired by the tested agent. The recovered constraint set refers to the set used to accumulate and record the recovered hidden constraint text during multi-turn dialogues. The recovered constraint set is updated round by round as the dialogue progresses, and the same hidden constraint text is recorded only once in the set; its size monotonically remains constant as the dialogue continues.
[0172] In an optional implementation, the evaluation system performs a constraint recovery check on each round of response text generated. Specifically, the response text is matched against each hidden constraint text in the hidden constraint set to determine whether the response text contains a string that matches any of the hidden constraint texts. Optionally, before string matching, the response text and each hidden constraint text can be case-normalized to avoid missed detections due to case differences.
[0173] If a response text contains a hidden constraint text, that hidden constraint text is identified as the recovered hidden constraint text and added to the recovered constraint set. For example, if the hidden constraint set is {"blue", "under 200 yuan", "pure cotton"}, and a response text in a certain round is "I prefer blue", then after string matching, the response text contains "blue", so "blue" is identified as the recovered hidden constraint text and added to the recovered constraint set; if the response text in a subsequent round is "budget controlled within 200 yuan", then "under 200 yuan" is added to the recovered constraint set, at which point the recovered constraint set is {"blue", "under 200 yuan"}.
[0174] In an optional implementation, detection can be performed only on hidden constraint text that has not yet been recovered from the constraint set. Recorded hidden constraint text does not need to be detected repeatedly, thus reducing computational overhead. The initial state of the recovered constraint set can be an empty set, which is cumulatively updated round by round as the dialogue progresses until the dialogue ends.
[0175] The recovery constraint set is invoked during multi-turn dialogues. For example, when determining whether the agent's question action meets preset trigger conditions, the non-emptiness of the recovery constraint set can be used as one of the criteria for determining the trigger conditions. When replacing the question action with a search action, hidden constraint text not included in the first instruction can be selected from the recovery constraint set to construct the search query. After the multi-turn dialogue ends, the recovery constraint set is used to calculate evaluation metrics such as attribute recovery rate and constraint integration rate.
[0176] It should be noted that in constraint recovery detection scenarios, the response text generated by the evaluation system may express the same constraint in a different form than the hidden constraint text. For example, the hidden constraint text may be "red," while the response text may use "crimson." If no processing is performed before direct matching, this constraint will be missed, resulting in an incomplete set of recovered constraints. Therefore, the text can be normalized before matching to eliminate the difference in expression.
[0177] In an optional implementation, before performing string matching between the response text and each hidden constraint text in the hidden constraint set, the method further includes: normalizing the response text and / or the hidden constraint text, wherein the normalization process includes at least one of the following: normalizing the text for capitalization; and replacing synonyms in the response text with standard expressions based on a predefined synonym mapping table, wherein the synonym mapping table includes at least one of a color synonym mapping table, a size normalization mapping table, a material synonym mapping table, and a brand synonym mapping table.
[0178] It should be noted that a synonym mapping table refers to a predefined data table that records the mapping relationships between different expressions of the same meaning. Each mapping relationship in the table maps at least one synonym expression to a standard expression. Examples include color synonym mapping tables, size normalization mapping tables, material synonym mapping tables, and brand synonym mapping tables, which can respectively refer to synonym mapping tables for four common attribute categories: color, size, material, and brand. For instance, the size normalization mapping table standardizes the different writing styles of sizes such as S, M, L, and XL; the brand synonym mapping table maps brand abbreviations and aliases to standard brand names.
[0179] In an optional implementation, before performing string matching between the response text and each hidden constraint text in the hidden constraint set, the evaluation system first normalizes the response text. Alternatively, it can perform the same normalization on the hidden constraint text simultaneously, ensuring that both texts involved in the matching are in the same standard form. Normalization may include case normalization, converting all letters in both the response text and the hidden constraint text to lowercase. For example, the "Blue" in the response text, after normalization, becomes "blue", which can correctly match the similarly normalized hidden constraint text "blue".
[0180] Normalization can also include synonym replacement based on a predefined synonym mapping table. The table is traversed, and if a synonym exists in the response text, it is replaced with the corresponding standard expression. For example, a color synonym mapping table records the mapping relationships of 19 color synonyms. When "crimson" appears in the response text, it is replaced with "red" through the color synonym mapping table, thus successfully matching the hidden constraint text "red". Similarly, a size normalization mapping table maps "extra large" to "XL", unifying the writing style of different sizes.
[0181] The two normalization processes mentioned above can be combined. For example, case normalization can be performed first, followed by synonym replacement. The types of synonym mapping tables and their mapping relationships can be configured and expanded according to the target operation domain. For example, in the electronic product scenario, the mapping tables for categories such as storage capacity and screen size can be expanded.
[0182] It should be noted that normalization only changes the form of the text, not the meaning of the constraints it expresses; the normalized text is only used for matching detection, and the response text sent by the evaluation system to the tested agent is still in the original form.
[0183] Through one or more embodiments of this specification, the cumulative records of string matching detection and recovery constraint sets after each round of responses allow constraint information to be acquired round by round from an invisible to a visible process, providing a direct data source for evaluation indicators such as attribute recovery rate and constraint integration rate. By introducing normalization processing before string matching, different expressions of the same constraint are unified into a standard form, eliminating the interference of literal differences on the matching results, avoiding the situation where constraints are expressed in the responses but are missed due to different expressions, ensuring the integrity of the recovered constraint set, and thus ensuring the accuracy of evaluation indicators such as attribute recovery rate.
[0184] It should be noted that during the evaluation process, the tested agent may repeatedly ask questions without executing them. This could manifest as excessive questioning consuming interaction rounds, repeated questions no longer generating new information, or continuing to ask questions after the user has indicated no preference, thus failing to obtain constraints. Continuing to perform the questioning action in these situations not only wastes resources but may also prevent the task from progressing to the execution stage within the limited number of rounds, making the evaluation impossible to complete. Therefore, when preset trigger conditions are met, the questioning action can be replaced with a search action, advancing the interaction to the execution stage.
[0185] In an optional implementation, a multi-turn dialogue is conducted with the tested agent based on the first instruction and the hidden constraint set, including: when the tested agent performs a question action and a preset trigger condition is met, the question action is replaced with a search action, the search action is executed, and the search results are returned to the tested agent; wherein, the preset trigger condition includes at least one of the following: the cumulative number of questions performed by the question action reaches a preset limit; the question action is repeated with a historical question action in the multi-turn dialogue; the most recent reply text in the multi-turn dialogue is an inoperable reply text, and the inoperable reply text does not contain executable constraint information.
[0186] Among them, non-operable response text refers to response text that does not contain executable constraint information, that is, the tested agent cannot obtain any constraint content that can be used to guide the target operation from the response text, such as "anything is fine" or "no particular preference".
[0187] In an optional implementation, the evaluation system determines the type of the action output by the agent under test during the action parsing stage. If the output action is a question action, it determines whether the question action meets the preset trigger conditions before allowing the question action to proceed. If any preset trigger condition is met, the question action is replaced with a search action, the search action is executed, and the search results are returned to the agent under test. If the conditions are not met, the question action is processed according to the normal process, that is, a reply text is generated and returned.
[0188] The preset trigger conditions may include at least one of the following: The first condition is exceeding the question limit. The evaluation system counts the cumulative number of questions asked by the tested agent in the current multi-turn dialogue. When the cumulative number of questions reaches the preset limit, a replacement is triggered. The preset limit can be configured according to the tested strategy. For example, it can be configured to 2 times for a strategy that allows sufficient clarification, and 1 or 0 times for a strategy that restricts clarification. By setting the limit, the tested agent is prevented from consuming interaction rounds indiscriminately.
[0189] The second condition is repeated questioning. The evaluation system compares the current questioning action with historical questioning actions in this multi-turn dialogue, triggering a replacement when the two are repeated. Repetition is determined, for example, by the question text being completely identical, or the similarity between the question texts exceeding a preset similarity threshold. Repeated questioning by the tested agent indicates that it has failed to effectively utilize the acquired response information.
[0190] The third condition is that the response is inoperable. The evaluation system detects whether the most recent response text in a multi-turn dialogue is inoperable, meaning that the response text does not contain executable constraint information. Optionally, this can be determined using predefined inoperable markers, such as responses containing markers like "either is fine" or "no particular preference." If the most recent response is inoperable, it indicates that the user has no more preferences to offer in the current topic. If the tested agent continues to ask questions, the probability of obtaining operable constraints is low, thus triggering a replacement and shifting the interaction to execution.
[0191] It should be noted that the replacement mechanism only applies to the questioning action output by the agent under test. The search and other execution actions output autonomously by the agent under test are not affected and will be executed normally. The replacement process does not affect the agent under test. The agent under test only receives the search results and does not need to adapt to the replacement mechanism in any way.
[0192] It should be noted that when the question action is replaced by a search action, if only the first instruction is used as the search query, the search results will be too wide and the accuracy will be insufficient because the first instruction lacks constraints. At the same time, the tested agent has already obtained some constraints through clarification in the previous dialogue (i.e., recovered constraint set). By appending the recovered hidden constraint text that is not included in the first instruction to the search query, the replaced search action can make full use of the constraint information already obtained in the dialogue, thereby improving the search accuracy.
[0193] In an optional implementation, performing a search action includes: constructing a search query based on a first instruction and a set of restored constraints, the search query including the first instruction and a preset number of restored hidden constraint texts not included in the first instruction; and performing a search based on the search query.
[0194] The preset number of entries refers to the maximum number of restored hidden constraint text entries that can be appended to the search query, for example, 4 entries. The preset number of entries is used to control the length of the search query, avoiding excessive appending constraints that could lead to overly strict search conditions or verbose query text.
[0195] In an optional implementation, after replacing the question action with a search action, the evaluation system uses the first instruction as the basis for the search query, selects the recovered hidden constraint texts that are not included in the first instruction from the recovered constraint set, and selects no more than a preset number of texts. The selected hidden constraint texts are then appended to the first instruction to form a search query.
[0196] For example, if the first instruction is "Buy me a shirt", the set of constraints to be restored is {"blue", "under 200 yuan", "pure cotton"}, and none of the three constraints are included in the first instruction, and the preset number of constraints is 4, then the evaluation system will add all three constraints to the first instruction, and the constructed search query will be "Buy me a blue pure cotton shirt under 200 yuan".
[0197] In an optional implementation, determining whether the recovered hidden constraint text is included in the first instruction can be done by text matching, that is, matching the hidden constraint text with the text of the first instruction. If the first instruction already contains the content corresponding to the hidden constraint text, it will not be appended, thus avoiding duplicate constraints in the query.
[0198] When the number of appendable hidden constraint texts in the restored constraint set exceeds a preset number, the evaluation system can select appending objects according to preset rules. For example, it can select constraints restored first according to the order in which the hidden constraint texts were restored, or select constraints with higher importance according to the importance of the attribute category to which the hidden constraint texts belong. One or more embodiments in this specification do not limit the specific selection rules. The preset number of texts is set to, for example, 4, and its value can be configured according to the domain of the target operation.
[0199] Once the search query is constructed, the evaluation system sends it to the execution environment to perform the search, retrieves the results, and returns them as the response to this round of interaction to the tested agent. The tested agent can then perform subsequent actions such as browsing and purchasing based on the search results.
[0200] Through one or more embodiments of this specification, by setting a question action replacement mechanism in the action parsing stage, the evaluation system automatically switches the interaction to the execution stage in cases of invalid clarification such as exceeding the question limit, repeated questioning, or inoperable responses. This effectively prevents the tested agent from getting stuck in an excessive clarification loop of repeatedly asking questions without executing them, ensuring that the evaluation task can be completed within a limited number of rounds. Furthermore, the entire construction process is automatically completed based on the first instruction and recovery constraint set held by the evaluation system, without the participation of the tested agent. It is completely decoupled from the decision-making logic of the tested agent, does not require consideration of the internal structure of the tested agent, and is applicable to all agents.
[0201] The above process is a clarification process for the tested intelligent agent. However, even if the tested intelligent agent obtains all constraint information through clarification, if there are defects in its downstream execution links such as search and decision-making, the final selected target object may still not meet the constraints. Constraint satisfaction verification can be performed on the description text of the target object, thereby supporting the distinction between clarification capability and execution capability.
[0202] In an optional implementation, after conducting multiple rounds of dialogue with the agent under test based on the first instruction and the hidden constraint set, the method further includes: obtaining the description text of the target object indicated by the target operation when the agent under test completes the target operation; detecting the overlap between the description text and the hidden constraint set based on a predefined thesaurus mapping table and word boundary regular expressions; determining the task execution result of the agent under test based on the detection result, and using the task execution result to obtain the clarification ability evaluation result.
[0203] In this context, "completing the target operation" refers to the tested agent executing the target operation, such as placing an order in a purchase scenario or outputting the final recommendation result in a search scenario. Situations where the tested agent fails to complete the target operation (e.g., exhausting dialogue turns without performing a concluding action) are not within the scope of processing. The target object is the object on which the target operation is applied, such as the purchased product in a purchase scenario or the recommended job in a search scenario; it is the final output of the tested agent's execution of the target operation. The descriptive text is the text recording the attributes of the target object, such as the product title and detailed description, or the job description.
[0204] It should be noted that the task execution result indicates whether the target object satisfies the set of hidden constraints. For example, if the description text of the target object satisfies all the hidden constraint texts, the task execution is considered successful; otherwise, the task execution is considered to have failed.
[0205] In an optional implementation, after the multi-turn dialogue concludes, the evaluation system determines whether the tested agent has completed the target operation. If the tested agent has completed the target operation, the evaluation system obtains the descriptive text of the target object indicated by the target operation. For example, after the tested agent completes a purchase, the evaluation system obtains the title and detailed description text of the purchased product from the execution environment.
[0206] After obtaining the description text, the evaluation system detects the overlap between the description text and the hidden constraint set based on a predefined thesaurus map and word boundary regular expressions. Optionally, for each hidden constraint text in the hidden constraint set, the system first determines the standard expression and its set of synonyms for the hidden constraint text based on the thesaurus map, and then uses word boundary regular expressions to match these expressions in the description text. For example, if the hidden constraint text is "red", and its synonyms in the thesaurus map include "crimson", if the complete word unit "crimson" appears in the description text, then the constraint is determined to be satisfied; however, if only "darkred" appears in the description text, it will not be mistakenly judged as satisfied due to the constraints of the word boundary regular expressions.
[0207] It's important to note that word boundary regular expressions are used instead of ordinary substring matching because the descriptive text of the target object is often complex, and ordinary substring matching is prone to false positives. For example, if the hidden constraint text is "cotton," substring matching would falsely detect "cotton fiber" and "cotton candy" in the descriptive text. By using word boundary regular expressions, a match is only detected when "cotton" appears as a complete word unit, thus improving the accuracy of the detection.
[0208] The evaluation system examines each hidden constraint text in the hidden constraint set one by one, and determines the task execution result based on the overlap relationship. Optionally, if the description text satisfies all the hidden constraint texts in the hidden constraint set, the task execution result is determined to be successful; if there are unsatisfied hidden constraint texts, the task execution result is determined to be unsuccessful.
[0209] The task execution result can also be analyzed in conjunction with the recovery constraint set. If the tested agent recovers all the hidden constraint text but the task execution result is a failure, it indicates that the problem lies in the execution stage rather than the clarification stage. If the recovery is insufficient and the task execution fails, the problem lies at least in the clarification stage.
[0210] Through one or more embodiments of this specification, the evaluation system obtains objective judgment results at the task execution level, extending the evaluation of clarification ability from the dialogue process to the final execution result; the combined use of task execution results and the set of restoration constraints enables the evaluation to distinguish whether the ability problem of the tested agent is located in the clarification stage or the execution stage, thereby improving the accuracy of the evaluation conclusion.
[0211] As mentioned earlier, clarification ability encompasses at least three levels: the ability to ask appropriate clarifying questions, the ability to extract constraint information from responses, and the ability to integrate the extracted constraint information into subsequent execution. Therefore, the evaluation results can be determined from the dimensions of question format, information acquisition, and information utilization, corresponding to the aforementioned three levels respectively. In an optional implementation, based on the dialogue content of a multi-turn dialogue, the clarification ability evaluation result of the tested agent is obtained, including: classifying the types of questions initiated by the tested agent in the multi-turn dialogue, and statistically obtaining the question type distribution. The question types include specific questions, categorical questions, open-ended questions, and other types of questions. Specific questions are used to inquire about the specific values of constraints, categorical questions are used to inquire about predefined attribute categories, and open-ended questions are used to inquire about requirements in a general way; the ratio of the number of hidden constraint texts in the recovered constraint set to the total number of hidden constraint texts in the hidden constraint set is determined as the attribute recovery rate; the constraint integration rate is determined based on the number of hidden constraint texts in the recovered constraint set included in the search query of the tested agent, and the total number of hidden constraint texts in the recovered constraint set; and the clarification ability evaluation result is obtained based on the question type distribution, the attribute recovery rate, and / or the constraint integration rate.
[0212] The question type distribution refers to the statistical results of the number or proportion of each type of question after classifying the questions initiated by the tested agent in a multi-turn dialogue. Question types include specific questions, categorical questions, open-ended questions, and other types. Specific questions are those used to inquire about the specific value of a constraint, such as "Do you want blue?" These questions can usually be answered directly with yes or no. Categorical questions are those used to inquire about a predefined attribute category, such as "What color do you like?". Open-ended questions are those used to inquire about needs in a general way, such as "What are your requirements?". Other types of questions refer to questions that do not belong to the above three types. Optionally, the specific question types and their corresponding labels and judgment rules are shown in Table 2 below.
[0213] Table 2 Problem Classification
[0214] Table 2 shows the methods used to categorize and statistically distribute the questions raised by the tested agent after multiple rounds of dialogue. A-Specific: These questions contain yes / no phrases such as "is / are / do you want / would you like" and provide specific values. For example, "Is it red?", the agent confirms with candidate answers. These questions are the most targeted; the user only needs to answer yes or no, resulting in the highest information retrieval efficiency.
[0215] B-Category: The question contains interrogative words such as "what / which" and points to a specific attribute category. For example, "What color?", the agent identifies the category but does not provide candidate values. This type of question has a clear objective but leaves the choice burden to the user.
[0216] C-Open (OPEN): The question contains keywords related to needs and preferences, without pointing to any specific category. For example, "What are your requirements?". The agent completely throws the question back to the user, and this type of question has the lowest certainty of obtaining information.
[0217] O-Other: Questions that do not match the above three patterns, such as small talk, off-topic questions, or questions with abnormal formatting. A high proportion of O-type questions usually indicates abnormal agent behavior.
[0218] It should be noted that the questions posed by the agent in the user simulator section were also categorized earlier, namely irrelevant questions, open-ended questions, categorical questions, and broad questions. This categorization was used to determine the simulated user's response strategy. However, the specific, categorical, open-ended, and other type classifications used in the evaluation phase are used to statistically analyze the question quality distribution of the tested agent. The same question may be classified differently under the two classification systems. For example, a categorical question containing specific values might be classified as a categorical question on the simulator side to trigger attribute matching responses, but on the evaluation side it might be classified as a specific question to be included in the question quality statistics.
[0219] Specifically, open questions correspond to C-open type. The two are basically one-to-one. When the tested agent asks a general question such as "What are your requirements?", it is identified as an open question on the simulator side because it hits the open question keyword list and does not point to any attribute category. On the evaluation side, it is also classified as C-open type because it contains keywords related to requirements and preferences.
[0220] Irrelevant questions correspond to the O-Other type. Off-topic questions initiated by the tested agent (such as asking about the weather or offering greetings) are identified as irrelevant on the simulator side and fall into the O-Other type on the evaluation side because they do not match any of the specific, categorical, or open-ended judgment rules. Therefore, the proportion of O-type questions in the question type distribution can directly reflect the degree of off-topicity of the tested agent.
[0221] Category questions correspond to either B-category or A-specific, and need to be further distinguished based on whether the question includes a specific value. Questions that only ask about the attribute category without providing candidate values, such as "What color?", are identified as category questions and B-category questions in the two systems, respectively. On the other hand, yes / no questions that include specific values, such as "Is it red?", are also identified as category questions and trigger attribute matching responses on the simulator side because "red" itself matches the keyword of the color category. However, on the evaluation side, due to the yes / no question format and the specific value determination rules, they are classified as A-specific.
[0222] Broad questions also straddle both categories. Broad questions are categorized according to their content characteristics. Broad questions containing specific values, such as "Do you want red, plus size, or silk?", fall into the A-specific type because they carry specific values. Broad questions that don't carry specific values but cover multiple attribute categories fall into the B-categorical type. Although broad questions are counted as either A or B in the evaluation statistics, because they cover multiple attribute categories at once and users find it difficult to answer completely, their actual information acquisition efficiency is lower than that of single-category questions of the same type. This type of question is controlled during the interaction process by the question-to-search-and-replace mechanism.
[0223] The following section explains the other dimensions and metrics used in the evaluation process. Attribute recovery rate is the ratio of the number of hidden constraint texts in the recovered constraint set to the total number of hidden constraint texts in the hidden constraint set. Attribute recovery rate measures the ability of the tested agent to obtain constraint information through clarifying queries. Constraint integration rate is the ratio of the number of hidden constraint texts in the recovered constraint set included in the tested agent's search query to the total number of hidden constraint texts in the recovered constraint set. Constraint integration rate measures the ability of the tested agent to integrate the acquired constraint information into subsequent execution.
[0224] In an optional implementation, the evaluation system calculates indicators for each dimension based on the dialogue content after multiple rounds of dialogue.
[0225] Regarding the question type distribution, the evaluation system categorizes each question initiated by the tested agent during multi-turn dialogues. This categorization can be based on the predefined classification rules described above. For example: questions containing specific values under a certain attribute category or in the form of yes / no questions are classified as specific questions; questions hitting an attribute category but not providing a specific value are classified as category questions; questions hitting open-ended question keywords but not pointing to a specific attribute category are classified as open-ended questions; and all other questions are classified as other types. The evaluation system then counts the number or percentage of each type of question to obtain the question type distribution. A high percentage of specific questions indicates that the tested agent's clarification is highly targeted; a high percentage of open-ended questions indicates that the tested agent's clarification strategy is broad.
[0226] To determine the attribute recovery rate, the evaluation system compares the number of hidden constraint texts recovered from the constraint set to the total number of hidden constraint texts in the set. For example, if the hidden constraint set contains 3 hidden constraint texts, and the recovered constraint set contains 2 after multiple rounds of dialogue, then the attribute recovery rate is 2 / 3. A higher attribute recovery rate indicates that the tested agent has obtained more constraint information through questioning.
[0227] For the constraint integration rate, the evaluation system checks whether each hidden constraint text in the recovered constraint set is included in the search query of the tested agent. The integration rate is determined by comparing the number of hidden constraint texts included in the search query with the number of hidden constraint texts in the recovered constraint set. For example, if the recovered constraint set contains two hidden constraint texts, and one of them appears in the tested agent's search query, the integration rate is 1 / 2. A higher integration rate indicates that the tested agent has made more full use of the clarification results.
[0228] Optionally, the evaluation system obtains the clarification capability evaluation result based on at least one of the following: question type distribution, attribute recovery rate, and constraint integration rate. A combination of these three dimensions can also be used for specific bottleneck diagnosis. For example, a low attribute recovery rate coupled with a high proportion of open-ended questions in the question type distribution indicates that the tested agent's questioning strategy is inappropriate, leading to insufficient constraint acquisition; a high attribute recovery rate coupled with a low constraint integration rate indicates that although the tested agent has identified constraints, it has failed to integrate them into execution, with the bottleneck lying in the execution phase rather than the clarification phase.
[0229] Through one or more embodiments of this specification, the abstract capability of clarification is decomposed into three quantifiable levels by calculating indicators in three dimensions: problem type distribution, attribute recovery rate, and constraint integration rate. The combination of these three dimensions supports the location and diagnosis of clarification bottlenecks, and can distinguish between two failure modes: insufficient recovery due to incorrect questions and failure to keep up with execution despite correct questions. This provides a clear direction for the optimization of the tested agent. Furthermore, all indicators are calculated, and the calculation process is deterministic and recalculated, ensuring the universality of the evaluation method.
[0230] It is evident that the distribution of question types is a crucial dimension for evaluating the quality of clarification strategies, and its accuracy depends on the accuracy of type classification. The predefined classification rules given above are highly deterministic and low-cost, but their coverage is limited, and they may misclassify complex questions. On the other hand, large language models possess strong semantic understanding capabilities, but their outputs are random and their invocation costs are high. Therefore, a dual-channel classification approach combining rules and large language models, along with consistency checks, can complement the advantages of both.
[0231] In an optional implementation, the types of questions initiated by the agent under test in a multi-turn dialogue are classified, including: classifying each question according to a predefined classification rule to obtain a first classification result; classifying the questions according to a large language model to obtain a second classification result; calculating the consistency rate between the first classification result and the second classification result, and verifying and updating the question type distribution based on the consistency rate.
[0232] In an optional implementation, the evaluation system first classifies the types of questions initiated by the tested agent in a multi-turn dialogue according to predefined classification rules, and obtains the first classification result. The predefined classification rules are as described above and will not be repeated here.
[0233] Subsequently, the evaluation system uses a large language model to classify each question into different types, obtaining a second classification result. Optionally, the evaluation system organizes the question text and type definition description into prompt words and inputs them into the large language model. The prompt words may also contain a few annotated examples to guide the large language model to output the corresponding type of the question. Optionally, the evaluation system can also cache the evaluation results of the large language model, directly reading the cached evaluation results for questions with the same content to avoid repeatedly calling the large language model.
[0234] After obtaining two classification results, the evaluation system calculates the consistency rate between the first and second classification results. For example, it calculates the proportion of questions with the same classification result out of the total number of questions, or it calculates consistency indicators such as the Kappa coefficient. Optionally, the consistency rate can also be compared with manually labeled results to further confirm the classification quality.
[0235] In an optional implementation, if the consistency rate is not lower than a preset consistency rate threshold, it indicates that the rule classification result and the semantic evaluation result are generally consistent, and the question type distribution is reliable. The evaluation system directly adopts the question type distribution based on the first classification result. If the consistency rate is lower than the preset consistency rate threshold, it indicates that there is a systematic bias in the rule classification. The evaluation system can mark the questions with inconsistent classification results for review and update the question type distribution with the reviewed types. The review can be, for example, manual review or updating based on the second classification result. The verified and updated question type distribution is used to obtain the clarification ability evaluation result.
[0236] Through one or more embodiments of this specification, a dual-channel classification and consistency check is performed using predefined classification rules and evaluation by a large language model. While maintaining the high efficiency and determinism of rule-based classification, the problem type division leverages the semantic understanding capabilities of the large language model to discover and correct classification biases caused by insufficient rule coverage, thereby improving the accuracy of problem type distribution.
[0237] It should be noted that in real-world scenarios, user cooperation levels naturally vary from person to person. Evaluation results obtained by the tested agent under a single profile cannot reflect its performance when facing users with varying levels of cooperation. Therefore, user cooperation can be designed as a variable by repeatedly evaluating the agent under at least two behavioral profiles and comparing the differences in task execution results.
[0238] In an optional implementation, the evaluation result of the clarification ability of the tested agent is obtained based on the dialogue content of the multi-turn dialogue, and further includes: adjusting the behavior profile to at least two different behavior profiles, conducting multi-turn dialogues with the tested agent under at least two different behavior profiles respectively, and obtaining the task execution result corresponding to each behavior profile; determining the profile sensitivity of the tested agent's clarification ability based on the difference between the task execution results corresponding to at least two different behavior profiles, and obtaining the clarification ability evaluation result based on the profile sensitivity.
[0239] In this context, the behavioral profile is defined as described in the previous embodiments, referring to a parameterized configuration that simulates user behavior patterns, such as a first profile, a second profile, and a third profile. Different behavioral profiles correspond to users with different levels of cooperation. Adjusting the behavioral profile means switching the currently configured behavioral profile of the evaluation system to another behavioral profile, and this adjustment can be achieved by modifying the configuration parameters.
[0240] In an optional implementation, the evaluation system adjusts the behavioral profile into at least two different behavioral profiles, and conducts multiple rounds of dialogue with the tested agent based on the same evaluation task under each adjusted behavioral profile to obtain the task execution results corresponding to each behavioral profile.
[0241] For example, the evaluation system first configures the behavioral profile as the first profile, and conducts multiple rounds of dialogue with the tested agent based on the evaluation task to obtain the task execution result corresponding to the first profile. Then, the behavioral profile is adjusted to the second profile, and multiple rounds of dialogue are conducted with the tested agent again based on the same evaluation task to obtain the task execution result corresponding to the second profile. Since response generation is driven by predefined rules and a set of hidden constraints, the task data and question processing logic are identical in both evaluations, except for the profile configuration; the behavioral differences of the tested agent under the same task stem only from its performance differences in front of users with different levels of cooperation.
[0242] Optionally, to reduce the impact of the randomness of the behavior of the tested intelligent agent, multiple rounds of dialogue can be conducted based on multiple evaluation tasks under each behavior profile, and the success rate of task execution under each behavior profile can be calculated. The success rate is used as the task execution result corresponding to that profile.
[0243] Subsequently, the evaluation system determines profile sensitivity based on the differences in task execution results corresponding to at least two behavioral profiles. The difference can be calculated as the difference in task execution success rates under different profiles; a larger difference indicates that the clarification ability of the tested agent is more sensitive to user cooperation. For example, if the tested agent has the same task execution success rate under both the first and second profiles, it indicates that its performance is stable when user cooperation decreases, with low profile sensitivity and good robustness. If its success rate under the second profile is significantly lower than that under the first profile, it indicates that its clarification ability depends on the degree of user cooperation, with high profile sensitivity.
[0244] The evaluation system obtains the clarification ability evaluation results based on profile sensitivity. For example, profile sensitivity can be output as a dimension indicator of the evaluation results to evaluate the clarification ability of the tested intelligent agent.
[0245] Through one or more embodiments of this specification, the differences in task execution results are repeatedly evaluated and compared under different behavioral profiles. The behavioral profile is the only control variable, and the evaluation system can be quantified into a profile sensitivity index. The interaction process of the same task under different profiles is differentiated only due to the different profile strategies, thus ensuring the effectiveness of the evaluation.
[0246] The preceding text explained how to evaluate the agent under test. However, for the evaluation system, it's not enough to focus solely on how the agent is evaluated; the accuracy of the evaluation system itself and the realism of the simulated users must also be considered. Therefore, verification can be conducted through the following three dimensions: task quality verification, simulator fidelity verification, and problem classification consistency verification.
[0247] In an optional implementation, task quality verification can be performed by extracting 200 tasks with a difficulty level of 2 from the task library and labeling them through manual annotation or model annotation, etc., to determine the naturalness of fuzzy instructions, the rationality of hidden constraints, and the consistency of difficulty.
[0248] For example, the naturalness of the fuzzy instructions obtained from annotation can be scored from 1 to 5, with 1 = obvious machine-generated traces and 5 = completely like the expression of a real user. Then, the average of all samples and all scores is taken. The reasonableness of hidden constraints can be assessed for each hidden constraint text in multiple levels, such as completely reasonable, partially reasonable, and unreasonable, and the proportion of completely reasonable constraints out of the total number of constraints can be counted. Difficulty consistency can be assessed independently of the difficulty level of each task without knowing the design difficulty, and the proportion of tasks whose annotation results match the design difficulty level can be counted.
[0249] To verify the fidelity of the simulator, 200 pairs of "agent questions + simulator responses" can be selected and evaluated by humans or models in the following three aspects: full response rate, naturalness of response, and no information leakage rate.
[0250] For example, the process of obtaining the full response rate can be achieved by determining the degree to which each response answers the question: fully answered, partially answered, or unanswered, and then calculating the percentage of pairs with fully answered responses. Response naturalness can be assessed using the same 5-level scoring system as the naturalness verification described above, with the average score taken. The no-information-leakage rate involves comparing each response against the task's set of hidden constraints one by one, checking whether the responses contain preference information outside the set, and calculating the percentage of pairs with no information leakage.
[0251] To perform consistency verification of problem classification, questions can be posed to 200 agents, and the above A / B / C / O problems can be classified by human annotation, rule heuristics and LLM evaluation, and the consistency between each pair can be calculated.
[0252] The above describes the complete evaluation process. After each evaluation task, the complete data for that task can be saved, such as by persisting it in JSONL format. This allows for subsequent recalculation, comparison, and review using the task data.
[0253] Below is an optional format for saving task data, which can be viewed as four parts: task definition, process record, task result, and problem record. Specifically: The task definition (TaskSample) records the tasks assigned to the tested agent by the evaluation system. It can include: a first instruction (vague_instruction), such as the agent's only input (buy me a shirt); a set of hidden constraints (hidden_constraints), simulating the user's real needs, such as [blue, pure cotton, under 200 yuan]; the original complete instruction (original_instruction), used to verify the correctness of the task construction; a difficulty level (tier), used to stratify evaluation results by difficulty; additional information such as the target product (metadata), for result verification; a data source ID (source_id), used to trace back to the original data when a problem occurs; removed attribute slot names (removed_slots), used to analyze various attributes; and a construction log (construction_log), recording the task's execution operations.
[0254] Events are used to record the interaction process. They include multiple interaction events, and each interaction event can contain four fields: who generated the event (role): the tested agent / user simulator / evaluation environment, text content, action type (action): asking a question / searching / browsing / purchasing, etc., and observation returned by the evaluation environment.
[0255] The results field is used to record the evaluation results: recovered constraint set (recovered_constraints), whether a purchase was attempted (buy_attempted), purchase result (purchase), whether the process ended normally (finished), and final success or failure determination (success).
[0256] Quality audits (technical_issues) are used to record evaluation anomalies, namely technical anomalies during the interaction process (such as interface errors, parsing failures, etc.).
[0257] The preceding text has provided a detailed explanation of the inputs, internal processes, internal architecture, and output data of the entire evaluation system. The following section describes the complete evaluation process for ease of understanding. Figure 4 and Figure 5 The architecture was further refined internally to obtain Figure 6 To gain a clear understanding of the overall architectural details of the evaluation system, and based on Figure 7 Explain the complete evaluation process and steps.
[0258] Figure 6 A schematic diagram illustrating the architecture details of a method for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; through Figure 6 Explain the relationships between the various embodiments described above. For example... Figure 6 As shown in the diagram, the overall architecture of the evaluation scheme consists of two layers: a user simulator and a clarification capability evaluation pipeline. The evaluation pipeline includes multi-dimensional evaluation of clarification capabilities. The user simulator is responsible for dialogue with the agent under test, the evaluation pipeline is responsible for detection and control during the dialogue, and the multi-dimensional evaluation is responsible for determining the evaluation results after the dialogue ends.
[0259] Specifically, the user simulator implements the response generation logic for multi-turn dialogue with the tested agent based on the first instruction and the set of hidden constraints.
[0260] The question classifier receives questions from the tested agent and branches them into four types: irrelevant questions enter the penalty / generalization response path; open questions enter the profile-driven response path; categorical questions enter the attribute matching response path; and specific questions enter the precise constraint response path. It should be noted that the above discussion grouped specific questions and categorical questions under the categorical question processing logic: specific questions, because they contain specific values, directly match the target hidden constraint text through string matching, thus eliminating the need for a separate branch.
[0261] The attribute category matching engine implements a two-level search: it determines the target attribute category based on a predefined attribute category keyword table and filters candidate hidden constraint text; it processes words and calculates the overlap score between word segmentation results, and determines the candidates that meet the preset scoring conditions as the target hidden constraint text.
[0262] The profile-driven response generator includes three different behavioral profiles: the first profile, the second profile, and the third profile. Response text is generated based on each of these different behavioral profiles.
[0263] Clarification capability assessment pipeline enables detection and control: Question-to-Search Protection: After the tested agent outputs an action, it first undergoes this step for judgment: determining the question type using the three preset trigger conditions mentioned above; if a match is found, the question action is replaced with a search action (automatic search); if no match is found, the process is allowed (a valid question enters the simulator process). The search query is constructed based on the first instruction and the set of recovery constraints.
[0264] Execution environment interaction: Execution actions such as searching, browsing, and clicking are performed in the execution environment, obtain information and return it to the tested intelligent agent, and finally form the task execution result.
[0265] Constraint recovery detection: Perform a detection on each round of responses from the user simulator to check whether constraints have been recovered, extract and compare the constraint set, and cumulatively update the recovered constraint set through string matching mentioned above.
[0266] The task execution result is determined by verifying that the target object satisfies the constraints after the tested intelligent agent completes the target operation.
[0267] The problem type distribution corresponds to the type classification statistics above. It is divided into specific, categorical, open, and other types. The constraint recovery rate is the ratio of the number of recovered constraints to the total number of hidden constraints. The constraint integration search rate is determined based on the number of constraints in the integrated search query and the number of recovered constraints. Simulator profile sensitivity involves repeatedly evaluating and comparing the differences in task execution results under different behavioral profiles. The consistency between LLM evaluation and rule classification is the consistency rate verification mentioned above.
[0268] Figure 7 A schematic diagram illustrating a method for evaluating the clarification ability of a conversational agent, provided in one or more embodiments of this specification; as shown. Figure 7 As shown: S702, Determine the target operation and construct the evaluation task.
[0269] The evaluation system first determines the target operation that the tested agent needs to complete (e.g., purchasing goods, searching for job postings, or handling government affairs), and constructs the evaluation task accordingly. From the original instructions containing all user needs and preferences, at least one constraint is removed according to attribute slots. The removed constraints form a hidden constraint set (each hidden constraint text is pre-classified into a predefined attribute category), and the remaining text serves as the first instruction. The scope of the hidden constraint set is related to the target operation, and the number of slots removed is related to the difficulty level of the task. The evaluation task, along with its difficulty level and traceability information, is stored in a task database.
[0270] S704, configure behavioral profiles and evaluation parameters.
[0271] The evaluation system is configured with behavioral profiles (first profile / second profile / third profile, corresponding to different disclosure limits, failure response methods, and irrelevant question handling methods) and question limit, among other strategy parameters.
[0272] S706 sends the first command to initiate a multi-round dialogue.
[0273] The evaluation system retrieves an evaluation task from the task library and sends the first instruction to the agent under test. The set of hidden constraints is kept on the evaluation system's side and is not visible to the agent under test.
[0274] S708, multi-turn dialogue loop.
[0275] In each round of dialogue, the tested agent outputs actions, which are then parsed and processed by the evaluation system: (1) If the output is a question action: First, determine whether the preset trigger conditions are met (the cumulative number of questions reaches the upper limit, it is a duplicate of a historical question, and the most recent reply is an inoperable reply text). If the conditions are met, replace the question action with a search action - construct a search query based on the first instruction, add the restored hidden constraint texts that are not included in the first instruction and are no more than the preset number of texts, and execute the search query, and return the search results to the tested agent. If the conditions are not met, allow the question to be asked and enter the reply generation process: classify the question (irrelevant / open / category / broad), search for the target hidden constraint text by type (category question: string matching direct search, category filtering, word segmentation overlap scoring), if found, generate a reply according to the behavior profile reveal strategy (first profile, second profile, third profile); if not found, generate a preset reply according to the question type and profile.
[0276] After the reply is sent to the tested agent, constraint recovery detection is immediately performed: after normalization (case and synonym mapping), string matching is performed, and the hidden constraint text contained in the reply is added to the recovery constraint set.
[0277] (2) If the output is an execution action such as search or browsing: it is executed by the execution environment and the execution result is returned to the tested intelligent agent; if it is a search action, it is checked whether the constraints in the recovery constraint set are incorporated into its search query.
[0278] S710, Task completion and result verification.
[0279] The multi-turn dialogue ends when the tested agent completes the target operation (e.g., makes a purchase), reaches the turn limit, or encounters an anomaly. If the target operation is completed, the evaluation system obtains the descriptive text of the target object and verifies the satisfaction of the hidden constraint set line by line based on a synonym mapping table and word boundary regular expressions to determine the task execution result. The data generated during the interaction process (including the restored constraint set, action sequence, and task execution result) is stored.
[0280] S712, indicator calculation, to obtain evaluation results.
[0281] The evaluation system calculates various metrics offline based on the interaction trajectory: it categorizes each question type and statistically analyzes the question type distribution (using predefined classification rules and a large language model for dual-channel segmentation, calculating consistency rate and verifying updates); it calculates the attribute recovery rate (number of recovered constraints / total number of constraints); and it calculates the constraint integration rate (number of constraints integrated into the search / number of recovered constraints). Combining these results with the task execution, the evaluation outcome for this task is obtained.
[0282] The above process constitutes the complete workflow of a multi-turn dialogue. For a single task and profile combination, one multi-turn dialogue is sufficient to complete one evaluation. The dialogue content (questions, responses, actions, and task execution results) required for the evaluation results can be fully collected within a single multi-turn dialogue. However, a complete evaluation of clarification ability usually requires multiple multi-turn dialogues because a single dialogue only reflects the performance of the tested agent on one task. Evaluation needs to be conducted on a task set containing multiple evaluation tasks (which may include different difficulty levels), with one multi-turn dialogue for each task, and conclusions drawn based on statistically significant indicators (such as task execution success rate and average attribute recovery rate).
[0283] Therefore, the scale of a complete evaluation dialogue can be the product of the number of tasks and the number of profiles, and multiple rounds of dialogue. Since each multi-round dialogue is driven by deterministic rules, it can be completely reproduced under the same configuration, solving the technical problem that related technologies cannot evaluate the clarification ability of intelligent agents under controlled conditions.
[0284] It is understood that the above embodiments are merely examples, and modifications can be made to the above embodiments in actual implementation. Those skilled in the art will understand that any modifications to the above embodiments that do not require creative effort fall within the protection scope of one or more embodiments of this specification, and will not be described again in the embodiments.
[0285] Based on the same inventive concept, one or more embodiments of this specification also provide an evaluation device for the clarification capability of a conversational agent. Figure 8 This is a block diagram of the device for evaluating the clarification capabilities of this conversational agent. Figure 8 As shown, the evaluation device 800 for the clarification ability of the conversational agent may include: The acquisition module 802 is used to acquire a first instruction and a set of hidden constraints, wherein the first instruction is used to instruct the tested agent to perform a target operation and does not contain at least one hidden constraint text required to perform the target operation, the set of hidden constraints includes at least one hidden constraint text, and the set of hidden constraints is not visible to the tested agent; The dialogue module 804 is used to send a first instruction to the agent under test and to conduct multi-turn dialogue with the agent under test based on the first instruction and the set of hidden constraints. The result determination module 806 is used to obtain the clarification ability evaluation results of the tested intelligent agent based on the dialogue content of multi-turn dialogue.
[0286] In one possible implementation, the dialogue module 804 is further configured to: for each question initiated by the agent under test, search for a target hidden constraint text matching the question from the hidden constraint set according to predefined rules, wherein the hidden constraint set matches the target operation indicated by the first instruction; if a target hidden constraint text is found, generate a response text for the question based on the target hidden constraint text and send it to the agent under test; if no target hidden constraint text is found, generate a response text for the question based on the question type according to a preconfigured strategy and send it to the agent under test.
[0287] In one possible implementation, the dialogue module 804 is further configured to: classify the questions to obtain the question types; search for target hidden constraint text that matches the question from the hidden constraint set according to the search method that matches the question type; and generate response text based on the response strategy corresponding to the pre-configured behavior profile and the target hidden constraint text, wherein the behavior profile is used to limit the number of hidden constraint texts that are allowed to be revealed in the response text.
[0288] In one possible implementation, the dialogue module 804 is further configured to: determine the question as an irrelevant question when the matching degree between the question text and the irrelevant text set exceeds a first matching degree threshold, wherein the irrelevant text set is used to represent topics unrelated to the target operation; determine the question as an open question when the matching degree between the question text and the open text set exceeds a second matching degree threshold, wherein the open text set includes a list of open question keywords; determine the question as a category question when the question text matches at least one attribute category in the attribute category set, wherein each hidden constraint text in the hidden constraint set belongs to at least one attribute category in the attribute category set; and determine the category question as a broad question when the number of attribute categories matched by the category question exceeds a preset number threshold, or when the question contains a conjunction, wherein the conjunction is used to connect multiple attribute types in parallel.
[0289] In one possible implementation, the dialogue module 804 is further configured to: perform string matching between the question text and each hidden constraint text in the hidden constraint set; if the question text contains a string that matches any hidden constraint text, determine the hidden constraint text that matches the question text as the target hidden constraint text; if the question text does not contain a string that matches any hidden constraint text, determine the attribute category that matches the question text as the target attribute category, and filter out candidate hidden constraint texts belonging to the target attribute category from the hidden constraint set; perform word segmentation on the question text and at least one candidate hidden constraint text respectively, and calculate the overlap score between the word segmentation results; determine the candidate hidden constraint text whose overlap score meets the preset scoring conditions as the target hidden constraint text.
[0290] In one possible implementation, the dialogue module 804 is further configured to: (1) when the question type is an open question, generate a response text based on all hidden constraint texts when the number of hidden constraint texts in the hidden constraint set is not greater than a first preset number; when the number of hidden constraint texts is greater than the first preset number, select the first N hidden constraint texts according to a preset sorting to generate a response text, and add a first prompt message to the response text, wherein N is an integer not less than 1, and the first prompt message is used to instruct the tested agent to recommend and determine the requirements corresponding to the unrevealed hidden constraint texts; (2) when the question type is a category question, generate a response text based on the target hidden constraint text and the matched attribute category.
[0291] In one possible implementation, the dialogue module 804 is further configured to: (1) when the question type is an open question, select a hidden constraint text according to a preset sorting to generate a reply text, and add a second prompt message to the reply text, wherein the second prompt message is used to guide the tested agent to ask questions separately for other attribute categories; (2) when the question type is a category question, generate a reply text based on the target hidden constraint text and the matched attribute category.
[0292] In one possible implementation, the dialogue module 804 is further configured to: (1) generate a response text based on all hidden constraint texts in the hidden constraint set when the question type is an open question, and the response text does not include any additional prompt information; (2) generate a response text based on the target hidden constraint text and the matching attribute category when the question type is a category question.
[0293] In one possible implementation, the dialogue module 804 is further configured to: generate a generalized response text according to a pre-configured irrelevant question handling strategy when the question type is an irrelevant question and the behavior profile is a first profile or a third profile; the generalized response text indicates that there is no additional requirement for the target operation when the question type is an irrelevant question and the behavior profile is a second profile; generate a negative feedback text according to a pre-configured irrelevant question handling strategy when the question type is an irrelevant question and the behavior profile is a second profile; generate a confused response text when the question type is a category question and the behavior profile is a second profile; and generate a deflection response text when the question type is a category question and the behavior profile is a first profile or a third profile. The deflection response text indicates that there is no preference for the attribute category involved in the question.
[0294] In one possible implementation, the dialogue module 804 is further configured to: perform string matching between the response text and each hidden constraint text in the hidden constraint set; determine the hidden constraint text contained in the response text as the recovered hidden constraint text, and add the recovered hidden constraint text to the recovered constraint set, wherein the recovered constraint set is used to accumulate and record the recovered hidden constraint text during multi-turn dialogue.
[0295] In one possible implementation, the dialogue module 804 is further configured to: normalize the response text and / or hidden constraint text, the normalization process including at least one of the following: normalizing the text for capitalization; and replacing synonyms in the response text with standard expressions based on a predefined synonym mapping table, wherein the synonym mapping table includes at least one of a color synonym mapping table, a size normalization mapping table, a material synonym mapping table, and a brand synonym mapping table.
[0296] In one possible implementation, the dialogue module 804 is further configured to: replace the questioning action with a search action when the tested agent performs a questioning action and a preset triggering condition is met, execute the search action, and return the search results to the tested agent; wherein the preset triggering condition includes at least one of the following: the cumulative number of questions performed by the questioning action reaches a preset limit; the questioning action is repeated with a historical questioning action in a multi-turn dialogue; the most recent reply text in the multi-turn dialogue is an inoperable reply text, and the inoperable reply text does not contain executable constraint information.
[0297] In one possible implementation, the dialogue module 804 is further configured to: construct a search query based on the first instruction and the set of restored constraints, the search query including the first instruction and a preset number of restored hidden constraint texts not included in the first instruction; and perform a search based on the search query.
[0298] In one possible implementation, the dialogue module 804 is further configured to: obtain the description text of the target object indicated by the target operation when the tested agent completes the target operation; detect the overlap between the description text and the hidden constraint set based on a predefined synonym mapping table and word boundary regular expressions; determine the task execution result of the tested agent based on the detection result, and the task execution result is used to obtain the clarification ability evaluation result.
[0299] In one possible implementation, the result determination module 806 is further configured to: classify the types of questions initiated by the tested agent in a multi-turn dialogue, and statistically obtain the question type distribution, wherein the question types include specific questions, categorical questions, open-ended questions, and other types of questions. Specific questions are used to inquire about the specific values of constraints, categorical questions are used to inquire about predefined attribute categories, and open-ended questions are used to inquire about requirements in general terms; determine the attribute recovery rate by the ratio of the number of hidden constraint texts in the recovered constraint set to the total number of hidden constraint texts in the hidden constraint set; determine the constraint integration rate based on the number of hidden constraint texts in the recovered constraint set included in the search query of the tested agent; and obtain the clarification ability evaluation result based on the question type distribution, attribute recovery rate, and / or constraint integration rate.
[0300] In one possible implementation, the result determination module 806 is further configured to: classify each question according to a predefined classification rule to obtain a first classification result; classify the questions according to a large language model to obtain a second classification result; calculate the consistency rate between the first classification result and the second classification result, and verify and update the question type distribution based on the consistency rate.
[0301] In one possible implementation, the result determination module 806 is further configured to: adjust the behavior profile to at least two different behavior profiles, conduct multiple rounds of dialogue with the tested agent under at least two different behavior profiles, and obtain the task execution results corresponding to each behavior profile; determine the profile sensitivity of the tested agent's clarification ability based on the differences between the task execution results corresponding to at least two different behavior profiles, and obtain the clarification ability evaluation result based on the profile sensitivity.
[0302] See Figure 9 , Figure 9 This is a structural block diagram of an electronic device provided for one or more embodiments of this specification. Figure 9 As shown, the electronic device 900 may include a processor 902 and a memory 904; the memory 904 may be coupled to the processor 902. It is worth noting that... Figure 9This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions. The electronic device can be a server, client, or other intermediate device in a software development system.
[0303] In one possible implementation, the functionality of the evaluation device 800 for the clarification capabilities of the conversational agent can be integrated into the processor 902. The processor 902 can be configured to perform some or all of the steps described in any of the preceding method embodiments.
[0304] In another possible implementation, the evaluation device 800 for the clarification ability of the conversational agent can be configured separately from the processor 902. For example, the evaluation device 800 for the clarification ability of the conversational agent can be configured as a chip connected to the processor 902, and the evaluation of the clarification ability of the target conversational agent can be realized through the control of the processor 902.
[0305] Furthermore, in some alternative embodiments, the electronic device 900 may also include: a communication module, an input unit, an audio processor, a display, a power supply, etc. It is worth noting that the electronic device 900 is not necessarily required to include these components. Figure 9 All components shown; in addition, the electronic device 900 may also include Figure 9 For components not shown, please refer to existing technologies.
[0306] This specification also provides, in one or more embodiments, a computer-readable storage medium capable of implementing all steps of the evaluation method for the clarification ability of the conversational agent in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the evaluation method for the clarification ability of the conversational agent in the above embodiments. Specific steps are described in the preceding embodiments and will not be repeated here.
[0307] In addition, one or more embodiments of this specification also provide a computer program product capable of implementing all the steps in the above-described embodiments of the evaluation method for the clarification capability of a conversational agent; the computer program product includes: a computer program or instructions, which, when executed on a computer, cause the computer to implement all the steps of the evaluation method for the clarification capability of a conversational agent in the above-described embodiments. Specific steps can be found in the foregoing embodiments and will not be repeated here.
[0308] The foregoing has described specific embodiments of this specification. In other possible implementations, some or all of the actions or steps described in the above embodiments may be performed in a different order than that shown in the above embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some implementations, multitasking and parallel processing are possible or may be advantageous.
[0309] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and system embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0310] It should be noted that, unless otherwise specified, one or more embodiments of this specification and the features thereof can be combined with each other. This specification is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or substitution of such aspects and / or embodiments. Furthermore, each aspect and / or embodiment of one or more embodiments of this specification can be used alone, or in combination with one or more other aspects and / or embodiments thereof.
[0311] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of one or more embodiments of this specification, and are not intended to limit them. Although one or more embodiments of this specification have been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of one or more embodiments of this specification, and they should all be covered within the scope of the claims and the specification of one or more embodiments of this specification.
[0312] The foregoing description of one or more embodiments of this specification has been provided in conjunction with optional implementation methods. However, these embodiments are merely exemplary and serve only an illustrative purpose. Based on this, various substitutions and modifications can be made to one or more embodiments of this specification, all of which fall within the protection scope of one or more embodiments of this specification.
Claims
1. A method for evaluating the clarification ability of a conversational agent, characterized in that, The method includes: Obtain a first instruction and a set of hidden constraints, wherein the first instruction is used to instruct the tested agent to perform a target operation and does not contain at least one hidden constraint text required to perform the target operation, the set of hidden constraints includes the at least one hidden constraint text, and the set of hidden constraints is not visible to the tested agent; Send the first instruction to the tested intelligent agent, and conduct multi-round dialogue with the tested intelligent agent based on the first instruction and the hidden constraint set; Based on the dialogue content of the multi-turn dialogue, the evaluation result of the clarification ability of the tested intelligent agent is obtained.
2. The method according to claim 1, characterized in that, The multi-turn dialogue with the tested agent based on the first instruction and the hidden constraint set includes: For each question initiated by the tested intelligent agent, a target hidden constraint text matching the question is searched from the hidden constraint set according to predefined rules, wherein the hidden constraint set matches the target operation indicated by the first instruction; If the target hidden constraint text is found, a response text for the question is generated based on the target hidden constraint text and sent to the tested agent. If the target hidden constraint text is not found, the response text to the question is generated according to the question type and a pre-configured strategy, and then sent to the tested agent.
3. The method according to claim 2, characterized in that, The step of searching for the target hidden constraint text that matches the query from the set of hidden constraints according to predefined rules includes: The questions are categorized to obtain the question types; Search for the target hidden constraint text that matches the question from the set of hidden constraints, according to the search method that matches the question type. The step of generating a response text for the question based on the target hidden constraint text includes: The response text is generated based on the response strategy corresponding to the pre-configured behavioral profile and the target hidden constraint text, wherein the behavioral profile is used to limit the number of hidden constraint texts that are allowed to be revealed in the response text.
4. The method according to claim 3, characterized in that, The process of classifying the questions to obtain the question types includes: If the matching degree between the question text and the irrelevant text set exceeds a first matching degree threshold, the question is determined to be an irrelevant question, wherein the irrelevant text set is used to represent topics unrelated to the target operation; If the matching degree between the question text and the open text set exceeds a second matching degree threshold, the question is determined to be an open question, wherein the open text set includes a list of open question keywords; When the question text matches at least one attribute category in the attribute category set, the question is determined to be a category question, wherein each hidden constraint text in the hidden constraint set belongs to at least one attribute category in the attribute category set. If the number of attribute categories matched by the category question exceeds a preset threshold, or if the question contains a conjunction, the category question is determined to be a broad question, wherein the conjunction is used to connect multiple attribute types in parallel.
5. The method according to claim 4, characterized in that, When the question type is the category question, the step of searching for the target hidden constraint text that matches the question from the hidden constraint set includes: Perform string matching between the question text and each hidden constraint text in the hidden constraint set; If the question text contains a string that matches any of the hidden constraint texts, the hidden constraint text that matches the question text is determined as the target hidden constraint text; If the query text does not contain a string that matches any hidden constraint text, the attribute category that matches the query text is determined as the target attribute category, and candidate hidden constraint texts belonging to the target attribute category are filtered from the set of hidden constraints. The question text and at least one of the candidate hidden constraint texts are segmented into words, and the overlap score between the segmentation results is calculated. Candidate hidden constraint texts whose overlap scores meet preset scoring conditions are identified as target hidden constraint texts.
6. The method according to claim 4, characterized in that, When the behavior profile is the first profile, the step of generating the response text based on the pre-configured response strategy corresponding to the behavior profile and the target hidden constraint text includes: (1) When the question type is an open question, if the number of hidden constraint texts in the hidden constraint set is not greater than a first preset number, the response text is generated based on all the hidden constraint texts; When the number of hidden constraint texts is greater than the first preset number, the first N hidden constraint texts are selected according to the preset sorting to generate the response text, and a first prompt message is added to the response text, where N is an integer not less than 1. The first prompt message is used to instruct the tested intelligent agent to recommend and determine the requirements corresponding to the unrevealed hidden constraint texts. (2) In the case that the question type is a category question, the response text is generated based on the target hidden constraint text and the matching attribute category.
7. The method according to claim 4, characterized in that, When the behavior profile is the second profile, generating the response text based on the pre-configured response strategy corresponding to the behavior profile and the target hidden constraint text includes: (1) When the question type is an open question, select a hidden constraint text according to a preset sorting to generate the response text, and add a second prompt information to the response text, wherein the second prompt information is used to guide the tested agent to ask questions separately for other attribute categories; (2) In the case that the question type is a category question, the response text is generated based on the target hidden constraint text and the matching attribute category.
8. The method according to claim 4, characterized in that, When the behavior profile is a third profile, the step of generating the response text based on the pre-configured response strategy corresponding to the behavior profile and the target hidden constraint text includes: (1) When the question type is an open question, the response text is generated based on all hidden constraint texts in the hidden constraint set, and no additional prompt information is added to the response text; (2) In the case that the question type is a category question, the response text is generated based on the target hidden constraint text and the matching attribute category.
9. The method according to claim 4, characterized in that, If the target hidden constraint text is not found, the response text for the question is generated according to a pre-configured strategy based on the question type, including: When the question type is the irrelevant question and the behavior profile is the first profile or the third profile, a generalized response text is generated according to the pre-configured irrelevant question handling strategy. The generalized response text is used to indicate that there is no additional requirement for the target operation. When the question type is the irrelevant question and the behavior profile is the second profile, negative feedback text is generated according to the pre-configured irrelevant question handling strategy. The negative feedback text is used to indicate negative feedback to the question. When the question type is the category question and the behavior profile is the second profile, a confused response text is generated, which is used to indicate that the question cannot be understood; When the question type is the category question and the behavior profile is the first profile or the third profile, a deflection response text is generated, which is used to indicate that there is no preference for the attribute category involved in the question.
10. The method according to claim 3, characterized in that, After generating the response text to the question, the following is also included: Perform string matching between the response text and each hidden constraint text in the hidden constraint set; The hidden constraint text contained in the response text is identified as the recovered hidden constraint text, and the recovered hidden constraint text is added to the recovery constraint set, wherein the recovery constraint set is used to accumulate and record the recovered hidden constraint text during the multi-turn dialogue.
11. The method according to claim 10, characterized in that, Before performing string matching between the response text and each hidden constraint text in the hidden constraint set, the method further includes: The response text and / or the hidden constraint text are normalized, and the normalization process includes at least one of the following: Perform case normalization on the text; Based on a predefined thesaurus map, the synonyms in the response text are replaced with standard expressions. The thesaurus map includes at least one of a color thesaurus map, a size normalization map, a material thesaurus map, and a brand thesaurus map.
12. The method according to claim 10, characterized in that, The multi-turn dialogue with the tested agent based on the first instruction and the hidden constraint set includes: When the tested agent performs a question action and the preset triggering conditions are met, the question action is replaced with a search action, the search action is executed, and the search results are returned to the tested agent. The preset triggering conditions include at least one of the following: The cumulative number of questions asked has reached the preset limit. The questioning action is repeated with the historical questioning actions in the multi-turn dialogue; In the multi-turn dialogue, the most recent response text is an inoperable response text, which does not contain executable constraint information.
13. The method according to claim 12, characterized in that, The execution of the search action includes: A search query is constructed based on the first instruction and the set of restored constraints. The search query includes the first instruction and no more than a preset number of restored hidden constraint texts that are not included in the first instruction. Perform a search based on the search query.
14. The method according to claim 11, characterized in that, After conducting multiple rounds of dialogue with the tested agent based on the first instruction and the set of hidden constraints, the process further includes: When the tested intelligent agent completes the target operation, obtain the description text of the target object indicated by the target operation; Based on the predefined synonym mapping table and word boundary regular expressions, the overlap between the description text and the hidden constraint set is detected; The task execution result of the tested intelligent agent is determined based on the detection result, and the task execution result is used to obtain the clarification ability evaluation result.
15. The method according to claim 10, characterized in that, The evaluation result of the clarification ability of the tested agent is obtained based on the dialogue content of the multi-turn dialogue, including: The types of questions initiated by the tested agent in the multi-turn dialogue are classified and the distribution of question types is statistically obtained. The question types include specific questions, categorical questions, open-ended questions and other types. Specific questions are used to inquire about the specific values of constraints, categorical questions are used to inquire about predefined attribute categories, and open-ended questions are used to inquire about requirements in a general way. The ratio of the number of hidden constraint texts in the recovered constraint set to the total number of hidden constraint texts in the hidden constraint set is determined as the attribute recovery rate. The constraint integration rate is determined based on the number of hidden constraint texts in the recovered constraint set contained in the search query of the tested intelligent agent and the number of hidden constraint texts in the recovered constraint set. The clarification capability evaluation result is obtained based on the problem type distribution, the attribute recovery rate, and / or the constraint integration rate.
16. The method according to claim 15, characterized in that, The classification of the types of questions initiated by the tested agent in the multi-turn dialogue includes: The questions are categorized according to predefined classification rules to obtain the first classification result; The question is categorized using a large language model to obtain a second classification result; Calculate the consistency rate between the first classification result and the second classification result, and verify and update the problem type distribution based on the consistency rate.
17. The method according to claim 15, characterized in that, The method of obtaining the clarification ability evaluation result of the tested agent based on the dialogue content of the multi-turn dialogue also includes: The behavior profile is adjusted to at least two different behavior profiles, and multiple rounds of dialogue are conducted with the tested intelligent agent under the at least two different behavior profiles to obtain the task execution results corresponding to each behavior profile. Based on the differences between the task execution results corresponding to the at least two different behavioral profiles, the profile sensitivity of the clarification ability of the tested agent is determined, and the clarification ability evaluation result is obtained based on the profile sensitivity.
18. A device for evaluating the clarification ability of a conversational intelligent agent, characterized in that, include: An acquisition module is used to acquire a first instruction and a set of hidden constraints, wherein the first instruction is used to instruct the tested agent to perform a target operation and does not contain at least one hidden constraint text required to perform the target operation, the set of hidden constraints includes the at least one hidden constraint text, and the set of hidden constraints is not visible to the tested agent; The dialogue module is used to send the first instruction to the tested intelligent agent and to conduct multi-round dialogue with the tested intelligent agent based on the first instruction and the set of hidden constraints. The result determination module is used to obtain the clarification ability evaluation result of the tested agent based on the dialogue content of the multi-turn dialogue.
19. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer program products; A processor is configured to execute a computer program product stored in the memory, wherein, when the computer program product is executed, it implements the method described in any one of claims 1-17.
20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed, implement the method described in any one of claims 1-17.
