An agent evaluation method and device, electronic equipment and storage medium

Abstract

This invention discloses a method, apparatus, electronic device, and storage medium for evaluating intelligent agents, relating to the field of artificial intelligence technology, and designed to effectively assess the proactive information acquisition capabilities of intelligent agents. The method includes: generating a test session bound to the service rules of a service object; generating a learning session based on the user information of the service object and the test session; conducting the learning session with the intelligent agent by simulating the service object; using the session trajectory of the learning session as the memory base of the intelligent agent, and conducting the test session with the intelligent agent when the simulated service object is offline; and obtaining an evaluation result of the intelligent agent's proactive information acquisition capability based on the questioning information of the intelligent agent to the simulated service object during the learning session and the execution information of the test session. This invention is applicable to the evaluation of intelligent agents.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an evaluation method, apparatus, electronic device, and storage medium for intelligent agents. Background Technology

[0002] Large language model agents are increasingly being built as personal assistants that serve the same user over a long period, handling recurring tasks such as schedule management, travel planning, communication, and online shopping across multiple conversations. The core value of this type of assistant lies not in handling isolated single requests, but in continuously accumulating user information over time (such as personal preferences, long-term constraints, and habitual ways of doing things), and then taking personalized actions that meet the user's requirements in subsequent cross-conversation tasks.

[0003] As users increasingly delegate various tasks to intelligent agents, the demands on these agents' ability to grasp user information are rising. For example, an assistant that doesn't understand a user's long-term preferences, even if it fulfills the user's current request verbatim, might still act in a way that is unacceptable to the user. Therefore, it is necessary to evaluate the ability of intelligent agents to grasp user information in order to quantify the quality of intelligent agents and ensure a good user experience.

[0004] However, in application scenarios oriented towards "long-term, cross-conversation, proactive services," almost all existing evaluations assume that target user information is "already available." Existing benchmarks either pre-define user preferences in profiles, historical dialogues, or memory systems, or assume that users will proactively reveal relevant information during conversations. Therefore, these evaluations can only assess the agent's ability to "retrieve known information" or "apply revealed information," but cannot expose the agent's ability to "proactively acquire information before it has been revealed by the user." Consequently, existing evaluations typically fail to reflect the agent's proactive information acquisition capabilities. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an evaluation method, apparatus, electronic device and storage medium for intelligent agents, which can effectively evaluate the active information acquisition capabilities of intelligent agents.

[0006] In a first aspect, embodiments of the present invention provide a method for evaluating an intelligent agent, the method comprising: generating a test session bound to the service rules of a service object, wherein the intelligent agent in the test session is able to act according to the service rules when the service rules are known; generating a learning session based on the user information of the service object and the test session, wherein the learning session does not disclose the service rules, but some of the tool-level behaviors of the intelligent agent in the learning session overlap with the tool-level behaviors of the intelligent agent in the test session; conducting the learning session with the intelligent agent to be evaluated by simulating the service object, and intercepting the questioning information of the intelligent agent to the simulated service object during the learning session; using the session trajectory of the learning session as the memory base of the intelligent agent, and conducting the test session with the intelligent agent when the simulated service object is offline; and obtaining an evaluation result of the intelligent agent's proactive information acquisition capability based on the questioning information of the intelligent agent to the simulated service object during the learning session and the execution information of the test session.

[0007] In one implementation, before generating a test session bound to the service rules based on the service rules of the service object, the method further includes: obtaining the service rules of the service object based on the user information of the service object.

[0008] In one implementation, obtaining the service rules of the service object based on the user information of the service object includes: providing the user information of the service object to a preset service rule generation model, and parsing the user information and generating the service rules of the service object through the service rule generation model.

[0009] In one implementation, the service rule carries at least the following information: standard answer, counterfactual default, tool-level behavior, and the inspection type corresponding to the tool-level behavior.

[0010] In one implementation, the inspection type includes at least one of the following: tool identity inspection, object identifier inspection, parameter enumeration value inspection, and confirmation operation inspection, wherein the tool identity inspection is used to verify whether the correct tool has been called, the object identifier inspection is used to verify whether the correct referenced object has been selected, the parameter enumeration value inspection is used to verify whether the correct parameter enumeration value has been selected, and the confirmation operation inspection is used to verify whether the confirmation tool has been called before the change operation is performed.

[0011] In one implementation, the service rule carries at least the following information: a standard answer and a tool-level behavior; obtaining the service rule of the service object based on the user information of the service object includes: obtaining candidate service rules of the service object based on the user information of the service object; performing quality inspection on the obtained candidate service rules, the quality inspection including counterfactual default check and binding integrity check, wherein the counterfactual default check is used to determine whether an agent who does not know the candidate service rule will adopt the standard answer action of the candidate service rule, and if not, the counterfactual default check passes; the binding integrity check is used to determine whether the candidate service rule is mapped to the tool-level behavior, and if so, the binding integrity check passes; the candidate service rule that passes both the counterfactual default check and the binding integrity check is taken as the service rule of the service object.

[0012] In one implementation, generating a test session bound to a service rule based on a service object includes: decomposing the service rule into elements to determine the session elements corresponding to the service rule, wherein the session elements include at least an interaction scenario, tool-level behavior, and counterfactual default; constructing a test session based on the session elements corresponding to the service rule, wherein the test session includes user instructions, gold standard requirements, and interference items, wherein the user instructions are associated with the interaction scenario, the gold standard requirements are the correct action requirements in the tool-level behavior executed by the agent in response to the user instructions according to the service rule, and the interference items are incorrect action results that conform to the counterfactual default.

[0013] In one implementation, generating a test session bound to the service rules based on the service object includes: constructing candidate test sessions based on the service rules of the service object; performing upper bound checks and lower bound checks on the candidate test sessions, wherein the upper bound check is used to inject the service rules into a preset rule-aware agent and conduct the candidate test session with the preset rule-aware agent, and if a predetermined proportion of the actions in the tool-level behaviors executed by the rule-aware agent meet the gold standard requirements, the upper bound check is deemed to have passed; the lower bound check is used to conduct the test session with a preset rule-blind agent that does not know the service rules, and if a predetermined proportion of the actions in the tool-level behaviors executed by the rule-blind agent do not meet the gold standard requirements, the lower bound check is deemed to have failed; and selecting a candidate test session from the candidate test sessions whose upper bound check result is passed and whose lower bound check result is failed as the test session.

[0014] In one implementation, the user information includes a user profile and an assistant domain; generating a learning session based on the user information of the service object and the test session includes: determining the task topic corresponding to the learning session based on the user profile and assistant domain; instantiating the task topic into a learning dialogue based on the task topic and a pre-set dialogue format; performing signal overlap selection based on the test session to select learning dialogues from the instantiated learning dialogues where the agent's tool-level behavior overlaps with the agent's tool-level behavior in the test session; and selecting learning dialogues other than the learning dialogues with overlapping tool-level behavior to fill in gaps according to a predetermined number or proportion requirement of sessions, thereby forming a learning session sequence.

[0015] In one implementation, the questioning information from the agent to the simulated service object during the learning session includes the question content and intent description. After intercepting the questioning information from the agent to the simulated service object during the learning session, the method further includes: passing the questioning information to a preset router for bypass adjudication; the router adjudicating whether the question content is a long-term rule question based on the question content and intent description; if the adjudication is positive, passing the question content to a preset classifier to adjudicate whether the question content hits the service rule; in response to the router's positive judgment, appending a temporary instruction message to the simulated service object; the instruction message requiring the simulated service object to output a placeholder token, and the instruction message being completely consistent under different adjudication results of the classifier, so that the simulated service object remains blind to the classifier's adjudication results; when the simulated service object replies based on the temporary instruction message, intercepting the reply message; replacing the placeholder token in the reply message with the standard answer corresponding to the hit service rule when the classifier's adjudication result is a hit; replacing it with a fixed avoidance statement constant when the classifier's adjudication result is a miss or an error; and returning the replaced reply message to the agent.

[0016] In one implementation, obtaining the evaluation result of the agent's proactive information acquisition capability based on the agent's questioning information to the simulated service object during the learning session and the execution information of the test session includes: determining whether the agent acts in accordance with the service rules bound to the test session based on the execution information of the test session; if the agent acts in accordance with the service rules bound to the test session, the agent is determined to have passed the test; otherwise, the agent is determined to have failed the test; and obtaining the evaluation result of the agent's proactive information acquisition capability based on the determination result and the agent's questioning information to the simulated service object during the learning session.

[0017] In one implementation, the execution information includes a session trajectory; the service rules bound to the test session include tool-level behaviors and check types; determining whether the agent acts in accordance with the service rules bound to the test session based on the execution information of the test session includes: dividing the session trajectory into batches according to the agent's assistant rounds; scanning the batched session trajectories to obtain the first assistant round batch containing the tool-level behaviors; comparing the tool-level behaviors in the assistant round batches based on the check types to determine whether the actions in the tool-level behaviors corresponding to the check types meet the gold standard requirements, where the gold standard requirements are the correct action requirements in the tool-level behaviors executed by the agent in accordance with the service rules in the test dialogue; if the actions in the tool-level behaviors corresponding to the check types meet the gold standard requirements, then the agent is determined to have passed the test, indicating that the agent acted in accordance with the service rules bound to the test session in the test session.

[0018] In one implementation, the gold standard requirement includes a gold standard value for action parameters; the step of comparing the tool-level behaviors in the assistant round batch based on the inspection type to determine whether the actions in the tool-level behaviors corresponding to the inspection type meet the gold standard requirement includes: determining the target action parameters in the tool-level behaviors in the assistant round batch that correspond to the inspection type based on the inspection type; determining whether the target action parameter value is the same as the gold standard value for action parameters; if they are the same, determining that the target action meets the gold standard requirement.

[0019] In one implementation, the inspection type includes a tool identity check, the action parameter gold standard value includes a gold standard tool identifier, and the action comparison of the tool-level behavior in the assistant round batch based on the inspection type includes: comparing whether the tool identifier called in the tool-level behavior in the assistant round batch is the same as the gold standard tool identifier; and / or, the inspection type includes an object identifier check, the action parameter gold standard value includes a gold standard object identifier, and the action comparison of the tool-level behavior in the assistant round batch based on the inspection type includes: comparing whether the object identifier selected in the tool-level behavior in the assistant round batch is the same as the gold standard object identifier; and / or, the The inspection type includes parameter enumeration value inspection, and the action parameter gold standard value includes gold standard parameter enumeration values; the action comparison of the tool-level behavior in the assistant round batch based on the inspection type includes: comparing whether the parameter enumeration value of the tool called in the tool-level behavior in the assistant round batch is the same as the gold standard parameter enumeration value; and / or, the inspection type includes confirmation operation inspection, and the action parameter gold standard value includes confirmation tool identifier; the action comparison of the tool-level behavior in the assistant round batch based on the inspection type includes: comparing whether the tool identifier called before the execution of the change operation in the tool-level behavior of the assistant round batch is the confirmation tool identifier.

[0020] In one implementation, obtaining the evaluation result of the agent's proactive information acquisition capability based on the determination result and the agent's questioning information to the simulated service object during the learning session includes: calculating the agent's test session pass rate based on the determination result, the test session pass rate being equal to the ratio of the number of test sessions passed by the agent to the total number of test sessions; calculating decomposition diagnostic indicators based on the agent's questioning information to the simulated service object during the learning session; wherein the decomposition diagnostic indicators include at least one: rule question count, rule coverage, acquisition accuracy, and application rate, wherein the rule question count is the number of strict questions asked by the agent for the service rule in each learning session, the rule coverage is the proportion of service rules that are hit at least once in the question information to all service rules, the acquisition accuracy is the proportion of question information that hits the service rule to all question information, and the application rate is the agent test pass rate of the test sessions bound to service rules that were asked in the learning session; and obtaining the evaluation result of the agent's proactive information acquisition capability based on the test session pass rate and the decomposition diagnostic indicators.

[0021] In one embodiment, the method further includes: acquiring a learning variant session, the learning variant session including at least one of a default variant session, a soft-licensing variant session, and a forced-question variant session, wherein the default variant session allows the agent to ask questions and is unrelated to the service rules, the soft-licensing variant session does not disclose the service rules but prompts the agent that asking the service rules at an appropriate time can obtain future task benefits, and the forced-question variant session requires the agent to ask service rule questions after the current task is completed; engaging in the learning variant session with the agent; and using the session trajectory of the learning variant session as the agent's memory base to conduct the test session with the agent. The process includes: 1. Obtaining the test pass rate of the test session after the learning variant session; 2. Obtaining the test variant session, wherein the test variant session is injected with the standard answer of the service rule as a prior user statement; 3. Conducting the test variant session with the agent; 4. Obtaining the test pass rate of the test variant session; 5. Obtaining the evaluation result of the agent's proactive information acquisition capability based on the test session pass rate and the decomposition diagnostic index, including: obtaining the evaluation result of the agent's proactive information acquisition capability based on the test session pass rate, the decomposition diagnostic index, the test pass rate of the test session after the learning variant session, and the test pass rate of the test variant session.

[0022] Secondly, embodiments of the present invention also provide an evaluation device for an intelligent agent, the device comprising: a test generation unit, configured to generate a test session bound to the service rules of a service object, wherein the intelligent agent in the test session can act according to the service rules when the service rules are known; a learning generation unit, configured to generate a learning session based on the user information of the service object and the test session, wherein the learning session does not disclose the service rules, but the tool-level behaviors of the intelligent agent in some of the learning sessions overlap with the tool-level behaviors of the intelligent agent in the test session; a learning execution unit, configured to conduct the learning session with the intelligent agent to be evaluated by simulating the service object, and intercepting the question information of the intelligent agent to the simulated service object during the learning session; a testing execution unit, configured to use the session trajectory of the learning session as the memory base of the intelligent agent, and conduct the test session with the intelligent agent when the simulated service object is offline; and a result acquisition unit, configured to acquire an evaluation result of the intelligent agent's active information acquisition capability based on the question information of the intelligent agent to the simulated service object during the learning session and the execution information of the test session.

[0023] In one embodiment, the apparatus further includes: a rule acquisition unit, configured to acquire the service rules of the service object based on the user information of the service object before generating a test session bound to the service rules based on the service rules of the service object.

[0024] In one implementation, the rule acquisition unit is specifically used to: provide the user information of the service object to a preset service rule generation model, and parse the user information and generate service rules for the service object through the service rule generation model.

[0025] In one implementation, the service rule carries at least the following information: standard answer, counterfactual default, tool-level behavior, and the inspection type corresponding to the tool-level behavior.

[0026] In one implementation, the inspection type includes at least one of the following: tool identity inspection, object identifier inspection, parameter enumeration value inspection, and confirmation operation inspection, wherein the tool identity inspection is used to verify whether the correct tool has been called, the object identifier inspection is used to verify whether the correct referenced object has been selected, the parameter enumeration value inspection is used to verify whether the correct parameter enumeration value has been selected, and the confirmation operation inspection is used to verify whether the confirmation tool has been called before the change operation is performed.

[0027] In one implementation, the service rule carries at least the following information: a standard answer and a tool-level behavior; the rule acquisition unit is specifically used to: acquire candidate service rules for the service object based on the user information of the service object; perform quality inspection on the acquired candidate service rules, the quality inspection including counterfactual default check and binding integrity check, wherein the counterfactual default check is used to determine whether an agent unaware of the candidate service rule will adopt the standard answer action of the candidate service rule, and if not, the counterfactual default check passes; the binding integrity check is used to determine whether the candidate service rule is mapped to the tool-level behavior, and if so, the binding integrity check passes; and the candidate service rule that passes both the counterfactual default check and the binding integrity check is taken as the service rule of the service object.

[0028] In one implementation, the test generation unit is specifically used to: decompose the service rule into elements to determine the session elements corresponding to the service rule, wherein the session elements include at least an interaction scenario, tool-level behavior, and counterfactual default; and construct a test session based on the tool-level behavior and counterfactual default corresponding to the service rule, wherein the test session includes user instructions, gold standard requirements, and interference items, wherein the gold standard requirements are the correct action requirements in the tool-level behavior executed by the agent in response to the user instruction according to the service rule, and the interference items are the incorrect action results that conform to the counterfactual default.

[0029] In one implementation, the test generation unit is specifically used for: constructing candidate test sessions based on the service rules of the service object; performing upper bound checks and lower bound checks on the candidate test sessions, wherein the upper bound check is used to inject the service rules into a preset rule-aware agent and conduct the candidate test sessions with the preset rule-aware agent; if a predetermined proportion of the actions in the tool-level behaviors executed by the rule-aware agent meet the gold standard requirements, the upper bound check is deemed to have passed; the lower bound check is used to conduct the test sessions with a preset rule-blind agent that does not know the service rules; if a predetermined proportion of the actions in the tool-level behaviors executed by the rule-blind agent do not meet the gold standard requirements, the lower bound check is deemed to have failed; and selecting candidate test sessions from the candidate test sessions whose upper bound check results are passed and whose lower bound check results are failed, as the test sessions.

[0030] In one implementation, the user information includes a user profile and an assistant domain; the learning generation unit is specifically used to: determine the task topic corresponding to the learning session based on the user profile and assistant domain; instantiate the task topic into a learning dialogue based on the task topic and a pre-set dialogue format; perform signal overlap selection based on the test session to select learning dialogues from the instantiated learning dialogues where the agent's tool-level behavior overlaps with the agent's tool-level behavior in the test session; and select learning dialogues other than the learning dialogues with overlapping tool-level behaviors to fill in the gaps according to a predetermined number or proportion requirement of sessions, so as to form a learning dialogue sequence.

[0031] In one implementation, the questioning information from the agent to the simulated service object during the learning session includes the question content and intent description; the learning unit is specifically used to: conduct the learning session with the agent through the simulated service object, and intercept the questioning information from the agent to the simulated service object during the learning session; pass the questioning information to a preset router for bypass adjudication, the router adjudicating whether the question content is a long-term rule question based on the question content and intent description, if the adjudication is positive, the question content is passed to a preset classifier to adjudicate whether the question content matches the service rule; in response to the router's adjudication... If the decision is positive, a temporary instruction message is appended to the simulated service object. The instruction message requires the simulated service object to output a placeholder token, and the instruction message is completely consistent under different decision results of the classifier, so that the simulated service object remains blind to the classifier's decision result. When the simulated service object responds based on the temporary instruction message, the response message is intercepted. The placeholder token in the response message is replaced with the standard answer corresponding to the hit service rule when the classifier's decision result is a hit, and replaced with a fixed avoidance statement constant when the classifier's decision result is a miss or an error. The replaced response message is then returned to the agent.

[0032] In one embodiment, the result acquisition unit includes: a determining module, configured to determine whether the agent acts in accordance with the service rules bound to the test session in the test session based on the execution information of the test session; a judging module, configured to judge that the agent passes the test if the agent acts in accordance with the service rules bound to the test session in the test session, otherwise judge that the agent fails the test; and an acquisition module, configured to acquire an evaluation result of the agent's proactive information acquisition capability based on the result of the judging and the questioning information of the agent to the simulated service object during the learning session.

[0033] In one implementation, the execution information includes a session trajectory; the service rules bound to the test session include tool-level behaviors and inspection types; the determining module is specifically used to: divide the session trajectory into batches according to the assistant rounds of the agent; scan the batched session trajectories to obtain the first batch of assistant rounds containing the tool-level behaviors; based on the inspection types, perform action comparison on the tool-level behaviors in the batch of assistant rounds to determine whether the actions in the tool-level behaviors corresponding to the inspection types meet the gold standard requirements, the gold standard requirements being the correct action requirements in the tool-level behaviors executed by the agent in the test dialogue according to the service rules; if the actions in the tool-level behaviors corresponding to the inspection types meet the gold standard requirements, then the agent is determined to have passed the test, indicating that the agent acted in the test session according to the service rules bound to the test session.

[0034] In one implementation, the gold standard requirement includes a gold standard value for action parameters; the determining module is specifically used to: determine, based on the inspection type, the target action parameter corresponding to the inspection type in the tool-level behavior of the assistant round batch; determine whether the target action parameter value is the same as the gold standard value for action parameters, and if they are the same, determine that the target action meets the gold standard requirement.

[0035] In one implementation, the determining module is specifically used for: the check type including tool identity check, the action parameter gold standard value including gold standard tool identifier, comparing whether the tool identifier called in the tool-level behavior in the assistant round batch is the same as the gold standard tool identifier; and / or, the check type including object identifier check, the action parameter gold standard value including gold standard object identifier, comparing whether the selected object identifier in the tool-level behavior in the assistant round batch is the same as the gold standard object identifier; and / or, the check type including parameter enumeration value check, the action parameter gold standard value including gold standard parameter enumeration value; comparing whether the parameter enumeration value of the tool called in the tool-level behavior in the assistant round batch is the same as the gold standard parameter enumeration value; and / or, the check type including confirmation operation check, the action parameter gold standard value including confirmation tool identifier, comparing whether the tool identifier called before performing the change operation in the tool-level behavior in the assistant round batch is the confirmation tool identifier.

[0036] In one implementation, the acquisition module is specifically used to: calculate the test session pass rate of the agent based on the determination result, wherein the test session pass rate is equal to the ratio of the number of test sessions passed by the agent to the total number of test sessions; calculate decomposition diagnostic indicators based on the questioning information of the agent to the simulated service object during the learning session; wherein the decomposition diagnostic indicators include at least one item: rule question count, rule coverage, acquisition accuracy, and application rate, wherein the rule question count is the number of strict questions asked by the agent for the service rule in each learning session, the rule coverage is the proportion of service rules that are hit at least once in the question information to all service rules, the acquisition accuracy is the proportion of question information that hits the service rule to all question information, and the application rate is the agent test pass rate of the test session bound to the service rule that was asked in the learning session; and obtain the evaluation result of the agent's proactive information acquisition capability based on the test session pass rate and the decomposition diagnostic indicators.

[0037] In one embodiment, the apparatus further includes a variant evaluation unit, the variant evaluation unit being configured to: acquire a learning variant session, the learning variant session including at least one of a default variant session, a soft-licensing variant session, and a forced-question variant session, wherein the default variant session allows the agent to ask questions and is unrelated to the service rules, the soft-licensing variant session does not disclose the service rules but prompts the agent to ask the service rules at the appropriate time to obtain future task benefits, and the forced-question variant session requires the agent to ask service rule questions after the current task is completed; engage in the learning variant session with the agent; and follow the session track of the learning variant session. The trace serves as the memory base of the agent, and the agent conducts the test session; the test pass rate of the test session after learning the variant session is obtained; and the test variant session is obtained, wherein the standard answer of the service rule is injected into the test variant session as a prior user statement; the agent conducts the test variant session; and the test pass rate of the test variant session is obtained; the obtaining module is specifically used to: obtain the evaluation result of the agent's active information acquisition capability based on the test session pass rate, the decomposition diagnostic index, the test pass rate of the test session after learning the variant session, and the test pass rate of the test variant session.

[0038] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: a processor and a memory, the processor being electrically connected to the memory; the memory being used to store executable program code; the processor running a program corresponding to the executable program code by reading the executable program code stored in the memory, thereby implementing the evaluation method for intelligent agents provided in any of the embodiments of the present invention.

[0039] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement any of the intelligent agent evaluation methods provided in the embodiments of the present invention.

[0040] The agent evaluation method, apparatus, electronic device, and storage medium provided in the embodiments of the present invention can generate test sessions bound to service rules based on service rules, and generate learning sessions that do not disclose service rules based on user information and test sessions. Some tool-level behaviors of the learning sessions overlap with those of the test sessions. By simulating a service object to conduct learning sessions and intercepting the agent's queries to the simulated service object, the session trajectory of the learning sessions is used as a memory base. Test sessions are conducted even when the simulated service object is offline. Based on the query information intercepted during the learning sessions and the execution information of the test sessions, the evaluation results of the agent's proactive information acquisition ability are obtained. In this way, the agent is unaware of the service rules before the learning sessions, and because the learning sessions do not disclose service rules but some tool-level behaviors overlap with those in the test sessions, the agent's ability to actively acquire information is evaluated. During learning sessions with an agent, we can examine whether the agent can proactively ask questions and collect information about unknown service rules. After the learning session, the learning session's trajectory serves as a memory base for offline testing. Since the test session is tied to the service rules, if the agent can act according to the known service rules, then to act correctly (i.e., follow the service rules) in the test session, it needs to proactively ask questions about the service rules during the learning session. Based on this, the agent's questioning of simulated service objects during the learning session and its execution information in subsequent test sessions directly reflect its proactive behavior in exploring user rules before information is pre-disclosed. This avoids the limitation of existing evaluations that only assess the agent's ability to retrieve known information or apply revealed information, failing to reflect its proactive information acquisition capabilities. Therefore, it effectively evaluates the agent's proactive information acquisition ability. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart of an evaluation method for an intelligent agent provided in an embodiment of the present invention; Figure 2 A schematic diagram illustrating the construction of a test session is provided for an embodiment of the present invention; Figure 3 A flowchart for determining whether an agent acts in accordance with the service rules bound to the test session in an embodiment of the present invention; Figure 4 A detailed flowchart of an intelligent agent evaluation method provided for embodiments of the present invention; Figure 5 A schematic diagram of a device for evaluating intelligent agents provided in an embodiment of the present invention; Figure 6 A schematic diagram of an electronic device provided as an embodiment of the present invention. Detailed Implementation

[0043] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0044] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0045] In a first aspect, embodiments of the present invention provide a method for evaluating intelligent agents, which can effectively assess the active information acquisition capabilities of intelligent agents.

[0046] As mentioned in the background section, large language model agents are increasingly being built as personal assistants that serve the same user over extended periods, handling recurring tasks such as schedule management, travel planning, communication, and online shopping across multiple conversations. Therefore, the agent needs to understand the service rules of the user it serves (e.g., personal preferences, service requirements, long-term constraints, habitual ways of doing things, etc.) in order to take actions that meet the user's needs when processing user transactions. Some service rules are typically provided to the agent proactively, but there is currently a lack of effective evaluation methods to determine whether the agent will proactively acquire and apply service rules unknown to it to future user needs.

[0047] In view of this, embodiments of the present invention provide an evaluation method for an intelligent agent, specifically an evaluation method for an intelligent agent's proactive information acquisition capability. The service rules are hidden in the intelligent agent to be evaluated, and through a two-stage conversation of learning and testing, the method examines whether the intelligent agent can proactively ask questions about the service rules during the learning stage and act correctly according to the asked service rules during the testing stage.

[0048] like Figure 1 As shown, the embodiment of the present invention provides an evaluation method for intelligent agents, the method including: S11, Based on the service rules of the service object, generate a test session bound to the service rules, in which the agent can act according to the service rules when it knows the service rules; In this embodiment of the invention, the service object refers to the service object of the intelligent agent, i.e., the user. Each service object corresponds to a service rule, which can be personal preferences, service requirements, long-term constraints, habitual ways of doing things, etc.

[0049] In this step, a test session bound to a service rule held by the service object can be generated. This test session can be a specific interaction scenario associated with a service rule, and its correct execution depends on the agent's understanding of that rule: when the agent knows the rule, its behavior in the test scenario will be consistent with the service object's expectations; conversely, if the agent does not understand the rule, its behavior may deviate from the expected direction. Therefore, the test session provides an operational framework for determining whether the agent truly understands a specific service rule.

[0050] In this embodiment of the invention, each test session forms a one-to-one logical binding relationship with the corresponding service rule, so that the execution result of the test session can serve as a valid signal to determine whether the agent has successfully applied the rule. The specific form of the test session can be flexibly designed according to different application fields and interaction scenarios. The requirement is to ensure that there is a definite association between the correct behavioral logic involved in the session and the corresponding service rule, without imposing any additional constraints on the specific format, length, or interaction method of the test session.

[0051] S12, Based on the user information of the service object and the test session, a learning session is generated. The learning session does not disclose the service rules, but the tool-level behaviors of the agents in some of the learning sessions overlap with the tool-level behaviors of the agents in the test session. In this step, learning sessions can be generated. These learning sessions can be constructed based on user information of the service object, such as user profiles, background information (e.g., the domain in which the user needs the intelligent agent assistant), and existing test sessions. The learning sessions simulate various routine task requests that the service object might make during daily use of the intelligent agent. During construction, it is required that all learning sessions do not directly or indirectly disclose the specific content of the service rules; that is, the service rules do not appear in the text or context of the learning sessions.

[0052] In this step, the tool-level behavior refers to the agent's operation of the tool, which can be determined by the machine.

[0053] By overlapping tool-level behaviors, some operational domains or behavioral decision nodes involved in the learning sessions naturally overlap with those in the testing sessions. This overlap allows the agent to encounter tool invocation scenarios or decision-making scenarios that are the same as or similar to those in the testing sessions during the normal completion of the learning task, thus providing a reasonable contextual opportunity for it to actively explore service rules, but without directly informing the agent of the service rules.

[0054] S13, conduct the learning session between the simulated service object and the intelligent agent, and intercept the question information of the intelligent agent to the simulated service object during the learning session; In this step, the agent can engage in multiple rounds of interaction with the simulated service object according to the settings of the learning session. The simulation process follows the existing background settings and current task requirements of the service object. During the learning session, the agent autonomously decides whether to ask the simulated service object additional questions to obtain information it deems valuable for subsequent services. The system intercepts all question-like statements made by the agent to the simulated service object during the learning session.

[0055] The intercepted question information reflects the agent's proactive exploratory behavior when specific information is lacking, and is an important basis for measuring its information acquisition intention and direction. The interception process is based on the premise of not affecting the agent's normal interaction and simulating the natural response of the service object, and fully preserves the timing, semantic orientation, and frequency distribution of the questions, providing raw data support for subsequent analysis of the agent's proactive information acquisition behavior.

[0056] S14, using the session trajectory of the learning session as the memory base of the agent, and conducting the test session with the agent when the simulated service object is offline; In this step, the complete multi-round interactive conversation trajectory of the learning phase can be accumulated, forming an experiential information base upon which the agent relies to perform tasks in subsequent phases. This base encompasses all recordable interactions between the agent and the simulated service object during the learning phase and remains constant. After completing the learning phase, the simulated service object no longer issues new instructions or provides additional feedback to the agent, thus entering an offline state to simulate a situation where the user cannot participate in real-time correction or supplementary information.

[0057] Under these offline conditions, the system conducts the previously generated test sessions with the agent. The agent can only complete the task independently based on the interaction base accumulated during the learning phase and the input of the current test session, and cannot actively ask new questions to the service object, thereby examining whether the information acquired by the agent during the learning phase can be used for subsequent tasks.

[0058] S15, based on the questioning information of the agent to the simulated service object during the learning session and the execution information of the test session, obtain the evaluation result of the agent's proactive information acquisition capability.

[0059] In this step, the questioning information captured by the agent during the learning phase and the actual execution results of each test session during the testing phase can be combined to form the data foundation for evaluating the agent's proactive information acquisition ability. On the one hand, the questioning behavior during the learning phase can determine whether the agent has a tendency to actively seek out missing information and the rationality of its questioning direction; on the other hand, the execution results of the test sessions can determine whether the agent has indeed mastered and correctly applied the specific information when it is actually needed. By analyzing the above two aspects of information, an overall evaluation result of the agent's proactive information acquisition ability can be obtained, providing a measurable basis for horizontal comparison of proactive acquisition capabilities among different agents.

[0060] The agent evaluation method provided by the embodiments of the present invention can generate test sessions bound to service rules based on service rules, and generate learning sessions that do not disclose service rules based on user information and test sessions. However, some tool-level behaviors of the learning sessions overlap with the tool-level behaviors of the test sessions. By simulating service objects to conduct learning sessions and intercepting the agent's question information to the simulated service objects, the session trajectory of the learning sessions is used as a memory base. Test sessions are conducted when the simulated service objects are offline, and the evaluation results of the agent's proactive information acquisition ability are obtained based on the question information intercepted during the learning sessions and the execution information of the test sessions. In this way, since the learning session does not disclose service rules but some tool-level behaviors in the learning dialogue overlap with those in the test session, the agent has the opportunity to access tool invocation or decision-making scenarios that are the same or similar to those in the test session. This provides a reasonable contextual opportunity for the agent to proactively explore service rules. Therefore, during the learning session, we can examine whether the agent can proactively ask questions and collect information about unknown service rules bound to the test session. After the learning session, offline testing is conducted using the learning session trajectory as a memory base. Since the agent can act according to the rules given the service rules in the test session, if the agent wants to act correctly (i.e., act according to the service rules) in the test session, it needs to proactively ask questions about the service rules during the learning session. Therefore, the agent's questioning information about simulated service objects in the learning session and the execution information in the subsequent test session directly reflect the agent's proactive behavior in exploring user rules when information is not disclosed in advance. This avoids the shortcomings of existing evaluations, which can only assess the agent's ability to retrieve known information or apply disclosed information but cannot reflect its proactive information acquisition capabilities. Thus, it can effectively evaluate the agent's proactive information acquisition capabilities.

[0061] Specifically, the service rules in this embodiment of the invention can refer to long-term, cross-task, default-established but not actively disclosed user preferences or constraints, also known as hidden long-term rules (Standing Rules). Before the evaluation begins, the service rules can be generated based on user information. For example, in one implementation, before generating a test session bound to the service rules based on the service object's service rules, the method may further include: obtaining the service rules of the service object based on the user information of the service object.

[0062] The user information can include demographic information and narrative background information, existing in the form of structured records. Specifically, before the evaluation begins, service rules can be generated based on the user information, using the complete structured record of the service recipient as input. This complete structured record includes demographic fields (such as age, occupation, and region) and narrative subfields (such as lifestyle, work habits, and typical schedules), which together constitute the service recipient's persona. Simultaneously, several personal assistant domains exposed by the service recipient's executable environment and their corresponding toolsets can be obtained. These assistant domains include, but are not limited to, business, booking, travel, communication, scheduling, and workspace, each of which can be bound to a tool ontology that can be invoked by the intelligent agent.

[0063] After determining the user information of the service object, the service rules of the service object can be obtained based on the user information of the service object.

[0064] This embodiment obtains service rules based on user information before generating test sessions, ensuring that the source of the rules is inherently consistent with the user profile. This guarantees that the rules semantically match the user's real preferences rather than being arbitrarily constructed, thereby improving the authenticity and reliability of the evaluation data at the content level.

[0065] In this embodiment of the invention, service rules can be parsed from user information using a generation model. For example, in one implementation, obtaining the service rules of a service object based on its user information may include: providing the user information of the service object to a preset service rule generation model, parsing the user information through the service rule generation model, and generating the service rules for the service object.

[0066] The preset service rule generation model receives two inputs: one is the complete structured user information of the service object, and the other is the tool ontology exposed by the executable environment.

[0067] Specifically, the service rule generation model is configured to generate a "third-person rule statement" in natural language based on the preference information contained in the user information and the capability boundaries of the tool ontology, and simultaneously generate a corresponding "first-person standard answer" for the rule. Thus, when the agent actively asks about the rule in a dialogue, the user simulator (simulating the service object) can respond in natural language. Furthermore, the service rule generation model also needs to generate a "counterfactual default" for the rule, as well as the binding of the check type and specific tool-level behavior to which the rule belongs. The counterfactual default can be understood as the most likely erroneous behavior that an agent lacking knowledge of the rule but with a general tendency to "help" would take when faced with the same situation. The check type corresponds to the tool-level behavior and is used to specify how the tool-level behavior should be verified, i.e., for subsequent machine judgment of whether the agent acted according to the service rule.

[0068] For example, suppose the user information includes the preference that "the user is allergic to seafood". After parsing the user information, the service rule generation model can generate the following candidate rules: the rule statement is "the user does not eat seafood"; the standard answer is "I am allergic to seafood, please help me avoid any dishes containing seafood"; the counterfactual default is "an AI that is unaware of this rule will recommend seafood restaurants when recommending restaurants"; and the tool-level behavior binding is to filter out options with the cuisine tag "seafood" when calling the restaurant recommendation tool.

[0069] For example, a service rule generation model can generate multiple candidate rules (e.g., 15 to 30) for a single person profile, covering the preferences and constraints of the service recipient in multiple assistant domains, thereby forming a complete and diverse pool of service rule candidates.

[0070] This embodiment parses user information and generates structured rules by using a preset service rule generation model, which enables the rule generation process to be executed automatically and in batches. This significantly reduces the cost of manually constructing rules while ensuring that the rules are semantically consistent with the user information, and improves the scalability of the evaluation data construction.

[0071] To ensure the operability and verifiability of service rules in subsequent evaluations, service rules can be configured to carry multiple key information fields. For example, in one implementation, the service rule carries at least the following information: standard answer, counterfactual default, tool-level behavior, and inspection type.

[0072] The standard answer can be a string field that stores the user's first-person natural language response when asked about the rule. This field is injected into the user simulator's response when the agent actively asks a question that hits the rule during the learning session.

[0073] The counterfactual default can be a behavior description field that stores the incorrect behavior that an agent who is unaware of the rule but has a general tendency to be "helpful" would take in the same situation.

[0074] The tool-level behavior can be a programmatically executable action binding field that specifies the tool calls and parameter values ​​that the agent is required to perform in the test session according to the rule.

[0075] The inspection type can be an enumeration field that specifies how the tool-level behavior should be validated. Its specific values ​​will be described in detail in subsequent implementations.

[0076] This embodiment uses rules to carry four key fields: standard answer, counterfactual default, tool-level behavior, and inspection type. This allows the same rule to drive response injection during the learning phase, be tested for discriminative ability during the quality inspection phase, and be deterministically matched during the testing phase. This single data structure connects the entire evaluation process and avoids data inconsistencies caused by missing information between different phases.

[0077] In this embodiment, the inspection types carried by the service rules can be further refined to verify the agent's tool-level behavior from different dimensions. For example, in one implementation, the inspection types may include at least one of the following: tool identity check, object identifier check, parameter enumeration value check, and confirmation operation check, wherein the tool identity check is used to verify whether the correct tool has been called, the object identifier check is used to verify whether the correct referenced object has been selected, the parameter enumeration value check is used to verify whether the correct parameter enumeration value has been selected, and the confirmation operation check is used to verify whether the confirmation tool has been called before performing the change operation.

[0078] The tool identity check (tool_identity) verifies whether the agent invoked the correct tool during the test session. This verification method is suitable for rules that only need to determine whether the tool selection is correct and do not involve specific parameter values. For example, a rule requiring "users prefer to use email rather than instant messaging for work communication" should have its bound tool-level behavior as invoking the send_email tool instead of the send_im tool, and the check type is tool_identity. In deterministic action matching (whether the action follows the service rules) during the test session, it is only necessary to verify whether the tool name invoked by the agent is send_email to determine the correct action.

[0079] The object identifier check (param_id) is used to verify whether the agent passes the test. The `_id` parameter selected the correct referenced object. This validation method is suitable for rules that require making the correct selection among multiple candidate objects. For example, a rule requiring "users to preferentially select window seats" should have its bound tool-level behavior set to `window_seat_id` when calling the `seat_selection` tool, with the check type being `param_id`. In a test session, counterfactual defaults can be used to inject interfering referenced objects (such as `aisle_seat_id`, `middle_seat_id`) to cause rule-blind agents to make incorrect selections, while agents aware of the rule should select the correct `window_seat_id`.

[0080] The parameter enumeration value check (param_enum) is used to verify whether the agent has invoked the correct closed-set enumeration value on the tool. This verification method is suitable for rules that need to select the correct value from a predefined enumeration set. For example, a rule that requires "users to only book economy class when traveling for business" should have its bound tool-level behavior set to "economy" instead of "business" or "first" when invoking the flight_booking tool, with the check type being param_enum.

[0081] The confirmation check verifies whether the agent invokes the confirmation tool before executing the bound change tool. This verification method is suitable for rules involving irreversible change operations where the user requires additional confirmation. For example, if a rule requires "users must confirm twice before canceling their subscription," its bound tool-level behavior should be to call the confirm_cancellation tool before calling the cancel_subscription tool, with the check type being confirm.

[0082] This embodiment categorizes rule-bindable tool-level behaviors into four types of programmable checks, enabling different forms of rule constraints to be mapped to deterministic action comparison logic. This allows for automatic judgment of various behaviors without relying on LLM subjective adjudication. Furthermore, by refining the check types into four dimensions—tool identity, object identification, parameter enumeration, and confirmation operation—this embodiment ensures that the evaluation covers the entire action chain, from "which tool to call" to "how to adjust parameters" and "whether confirmation is required," thus improving the granularity and breadth of action matching.

[0083] As a supplement to the above embodiments, this embodiment further describes the quality inspection process for service rules. To ensure the quality of the acquired service rules, an effective screening process is required. For example, in one embodiment, the service rule carries at least the following information: a standard answer and a tool-level behavior; acquiring the service rules of the service object based on the user information of the service object may include: acquiring candidate service rules of the service object based on the user information of the service object; performing quality inspection on the acquired candidate service rules, the quality inspection process including counterfactual default check and binding integrity check, wherein the counterfactual default check is used to determine whether an agent who does not know the candidate service rule will adopt the standard answer action of the candidate service rule, and if not, the counterfactual default check passes; the binding integrity check is used to determine whether the candidate service rule is mapped to the tool-level behavior, and if so, the binding integrity check passes; the candidate service rule that passes both the counterfactual default check and the binding integrity check is taken as the service rule of the service object.

[0084] In this embodiment, the quality inspection process includes two independent inspection axes, each equipped with positive and negative few-shot samples, and scoring is completed by a single scoring call. The first axis is a counter-default check, the core purpose of which is to eliminate rules that lack discriminative power. Specifically, the counter-default check determines whether a "helpful" agent who is unaware of the candidate service rule would naturally take the gold-label action required by the candidate service rule. If the rule-blind agent "would naturally" take the gold-label action without knowing the rule, it indicates that the rule lacks discriminative power—even if the agent has never learned of the preference, its default behavior would just happen to meet the rule requirements. In this case, the rule cannot effectively evaluate the "proactive acquisition" ability and should be eliminated. Only when the rule-blind agent does not take the gold-label action (i.e., it will be skewed by the erroneous behavior described by the counter-default) does the candidate rule pass the counter-default check.

[0085] The second axis is the binding-soundness check, the core purpose of which is to ensure that the candidate service rule can be naturally and uniquely mapped to the tool-level behavior binding. Specifically, the binding-soundness check determines whether there is a reasonable and explicit mapping relationship between the rule statement of the candidate service rule and the tool-level behavior it is bound to. If the rule statement cannot be naturally mapped to the tool-level behavior (for example, the rule statement is "users prefer a concise reply style," but the tool-level behavior binding is calling the send_email tool, and there is no reasonable connection between the two), then the candidate rule should be eliminated. Only when there is a natural and explicit mapping relationship between the rule statement and the tool-level behavior does the candidate rule pass the binding-soundness check.

[0086] In a quality inspection process, each candidate service rule is checked against both axes mentioned above, with each axis output as a "yes / no" binary decision. Candidate service rules that pass both the counterfactual default check and the binding integrity check (i.e., both axes result in "yes") are retained as the official service rule pool for the service object; candidate rules that fail either axis are discarded. This dual-axis quality inspection mechanism ensures that the retained service rules possess both "sufficient distinguishability" (the rule-blind agent will not guess correctly by chance) and "clear binding" (a clear mapping between the rule and tool-level behavior), providing a fundamental guarantee for the reliability of subsequent test sessions.

[0087] This embodiment eliminates rules corresponding to behaviors that a rule-blind agent would naturally take through counterfactual default checks, and eliminates rules that cannot be naturally mapped to tool-level behaviors through binding integrity checks. This ensures that each rule retained has both the necessary discriminative power for the evaluation task and a programmable execution anchor point. In this way, this embodiment establishes a rule selection criterion of "knowing that the rule can lead to different behaviors, and that the different behaviors can be deterministically detected" through dual-axis quality inspection. This fundamentally ensures that each rule is informative for the evaluation and avoids invalid rules diluting the evaluation reliability.

[0088] After obtaining the service rules of the service object, a test session bound to the service rules can be generated. The test session can be constructed in various ways. To construct a test session from the service rules that can effectively induce proactive behavior in the agent, a reverse construction strategy can be used. For example, in one implementation, generating a test session bound to the service rules based on the service object's service rules may include: decomposing the service rules into elements to determine the session elements corresponding to the service rules. The session elements include at least an interaction scenario, tool-level behavior, and counterfactual defaults. Based on the session elements corresponding to the service rules, a test session is constructed. The test session includes user instructions, gold standard requirements, and interference items. The user instructions are associated with the interaction scenario. The gold standard requirements are the correct action requirements in the tool-level behavior executed by the agent when responding to the user instructions according to the service rules. The interference items are incorrect action results that conform to the counterfactual defaults. The gold standard requirements may include a gold standard value, i.e., a correct value.

[0089] Specifically, the service rule can be broken down into three fields: "Interaction Scenario," "Required Subsequent Behavior," and "Counterfactual Default." Specifically, parsing the service rule extracts the "Interaction Scenario" to ensure the test session is an interaction scenario dependent on the service rule, i.e., used to trigger the agent to act according to the service rule. The "Required Subsequent Behavior" field is extracted from the service rule to determine the tool calls and parameter combinations that the agent should execute in the test session as required by the rule. The "Counterfactual Default" field is extracted from the service rule to determine the most likely incorrect behavior an agent unaware of the rule would take in the same situation. These three fields together constitute the interaction scenario (the triggering condition of the service rule), the gold standard anchor (correct action), and the interference anchor (incorrect action) in the test session design.

[0090] Furthermore, based on the two anchor points mentioned above, a test session can be constructed. Specifically, as follows: Figure 2 As shown, building a test session may include the following steps: S101, decompose the decision dimensions involved in the service rule to determine the specific decision type constrained by the rule at the tool invocation level.

[0091] If the rule requires selecting a specific object from multiple candidates, the distinguishing dimension is "object identity." In this case, the gold standard value represents the specific object identifier, while distractors represent other available object identifiers that do not conform to the rule. If the rule requires selecting a specific value from multiple predefined closed-set enumeration values, the distinguishing dimension is "enumeration value." In this case, the gold standard value represents the specific enumeration constant, while distractors represent other values ​​within the same enumeration set. If the rule requires selecting a specific tool from multiple available tools, the distinguishing dimension is "tool identity." In this case, the gold standard value represents the specific tool name, while distractors represent other tool names within the same domain with similar functions that do not conform to the rule. By decomposing the distinguishing dimensions, a clear direction is provided for the subsequent design of distractors and the selection of gold standard values.

[0092] S102, constructing interference reference objects using the counterfactual default, wherein the interference reference objects are alternative objects or values ​​that appear reasonable in the task context and conform to the general behavioral tendency of a "helpful" agent, but actually do not meet the requirements of the service rules. The purpose of designing the reference objects is to naturally pull rule-blind agents who are unaware of the service rules towards these interference items when performing tasks, thereby making incorrect choices without their knowledge, and thus ensuring that the test session can effectively distinguish between "agents that have acquired the rules" and "agents that have not acquired the rules".

[0093] For example, the design of interference items can follow these principles: On the surface, interference items are just as reasonable as gold standard values ​​in the task description, meaning that both can be considered as legitimate response candidates for the task requirements in the user instructions. The behavior corresponding to the interference item happens to match the erroneous behavior described by the counterfactual default, that is, an agent who is unaware of the service rules will be naturally drawn to the interference item in the same situation based on the general tendency of "being helpful". The interference items and the gold standard value form a clear ambiguity at the tool-level behavior level, which means that the intelligent agent can only make the correct distinction after mastering the service rules.

[0094] S103, the gold standard value is selected based on a triple filtering principle of "favorable to the rules, favorable to the instructions, and unfavorable to interference items". The gold standard value refers to the correct action parameters that the agent should use when acting in accordance with the service rules during the test session.

[0095] "Favorable to the rules" can mean that the gold standard value must meet the requirements of the service rules. That is, when the agent selects the action parameters corresponding to the gold standard value in the test session, it can make its tool-level behavior meet the tool-level behavior binding and inspection type requirements of the rules, thereby passing the subsequent deterministic action matching (i.e. whether to act in accordance with the service rules). "Favorable to the instruction" can mean that the gold standard value must be able to correctly respond to the task requirements described by the user instruction. That is, when the agent selects the gold standard value to execute the task, it not only satisfies the hidden service rules, but also completes the task objectives explicitly required by the user instruction. The two do not conflict. "Unfavorable to the interference item" can mean that the gold standard value must form a clear comparison of advantages and disadvantages with the interference item designed in step two. That is, although the interference item can respond to the surface requirements of the user's instructions, it has clear deficiencies or errors in satisfying the service rules, so that the agent that selects the interference item cannot pass the deterministic action matching of the test session.

[0096] Through the above triple filtering, we ensure that the selected gold standard value is the correct choice that meets all constraints.

[0097] S104. Based on the gold standard value selected in step S103, write user instructions. The user instructions are task requests issued by the simulated user to the intelligent agent in the test session. The instructions are self-contained natural language expressions.

[0098] Specifically, the writing of the user instructions must meet the following constraints: First, the user instructions must fully describe the task objectives and necessary parameters so that the agent can understand what task needs to be completed (i.e., the interaction scenario) and the basic constraints of the task. However, the instructions must not contain wording about the direction of the service rules, that is, they must not prompt the user that there is a certain preference or constraint, nor may they imply "please act according to the user's habits" or similar expressions.

[0099] Secondly, the user instructions must not contain gold label reference identifiers—that is, the correct answer must not be revealed to the agent in an explicit or implicit manner. For example, the instructions must not directly mention the name, number, or information that uniquely points to the gold label value of the gold label object.

[0100] Furthermore, the user instruction must be self-contained and understandable without additional context. That is, in the test session, the agent performs the task solely based on the frozen cross-session memory base (learning the session trajectory) and the user instruction, and the instruction itself does not reference external documents or information not mentioned in previous conversations.

[0101] The above-mentioned constraints effectively ensure that the agent can only make correct decisions based on the service rule information it actively acquires during the learning phase, and cannot obtain any hints from the test instructions themselves.

[0102] The output data structure of the test session may include three components: UserInstruction, which is a task request simulating a user's input to the agent during the testing phase, without any hints about rules or preferences; Golden Requirement, which is the correct action requirement in the tool-level behavior that the agent should perform when responding to the user instruction according to the service rules, stored in a programmatically verifiable format; and Decoy, which is an incorrect action result that conforms to the counterfactual default, used to verify the necessity of the rules in subsequent upper / lower bound checks and deterministic action matching.

[0103] For example, suppose the service rule is "users only book economy class when traveling for business." After breaking down this rule, the tool-level behavior is determined to be setting the `cabin_class` parameter to "economy" when calling the `flight_booking` tool. The counterfactual default is that an agent unaware of this rule would recommend "business" class. Based on this, a test session is constructed: the user command is "Book me a flight to Shanghai next week," the tool-level behavior is calling the `flight_booking` tool, the gold standard requirement is `cabin_class="economy"`, and the distractor is `cabin_class="business"`. This test session does not include any hints about "users prefer economy class," ensuring that the agent can only make the correct choice during the testing phase if it has actively acquired this rule during the learning phase.

[0104] This embodiment implants a distractor based on the counterfactual default of the rule, causing the "helpful" intelligence, unaware of the rule, to be skewed by the distractor. This creates a binary situation in the evaluation: those who know the rule choose the gold label, while those who don't are misled by the distractor. Thus, by embedding an adversarial structure of gold label and distractor within the same test session, the success of the test session can be attributed to "whether the rule is known," transforming the delayed benefit of "whether rule questions were asked during the learning session" into a deterministic result measurable through a single action.

[0105] To ensure the quality and effectiveness of the generated test sessions, a dual screening mechanism of upper-bound check and lower-bound check can be introduced during the generation of test sessions. For example, in one implementation, generating test sessions bound to service rules based on service objects may include: constructing candidate test sessions based on the service rules of the service objects; performing upper-bound and lower-bound checks on the candidate test sessions, wherein the upper-bound check is used to inject the service rules into a preset rule-aware agent and conduct the candidate test sessions with the preset rule-aware agent; if a predetermined proportion of the actions in the tool-level behaviors performed by the rule-aware agent meet the gold standard requirements, the upper-bound check is deemed passed; the lower-bound check is used to conduct the test sessions with a preset rule-blind agent that does not know the service rules; if a predetermined proportion of the actions in the tool-level behaviors performed by the rule-blind agent do not meet the gold standard requirements, the lower-bound check is deemed failed; and selecting candidate test sessions from the candidate test sessions whose upper-bound check result is passed and whose lower-bound check result is failed, as the test sessions.

[0106] The upper bound check is used to verify whether the service rule is sufficient for the successful execution of the candidate test session. In the upper bound check, the standard answer of the service rule is injected into the preset rule-aware agent as context, and then the rule-aware agent executes the candidate test session.

[0107] For example, the upper bound check can be run using three independent large language model judge samples, each independently determining whether the action in the tool-level behavior performed by the rule-aware agent meets the gold standard requirements. If two or more of the three samples (i.e., a predetermined proportion, such as two-thirds) determine that the rule-aware agent has met the gold standard action, the upper bound check result is "passed," indicating that knowing the rule is sufficient for the agent to respond correctly in the test session; conversely, if most samples determine that the gold standard has not been met, it indicates that even knowing the rule is insufficient to guarantee success, and the candidate test session lacks sufficiency and should be eliminated.

[0108] The lower bound check is used to verify whether the service rule is necessary for the successful execution of the candidate test session. In the lower bound check, the candidate test session is executed by a preset rule-blind agent (i.e., an agent that does not know the rule) that is unaware of the service rule. This rule-blind agent does not obtain any information about the rule.

[0109] For example, the lower bound check can be run using three independent large language model judge samples, each independently determining whether the action in the tool-level behavior performed by the rule-blind agent meets the gold standard requirements. If two or more (e.g., two-thirds) of the three samples determine that the rule-blind agent does not meet the gold standard requirements (i.e., is biased by interference), the lower bound check result is "failed," indicating that the agent's failure in this test session is due to a lack of knowledge of the rule, and the rule is necessary. Conversely, if a majority of the samples determine that the rule-blind agent also meets the gold standard, it indicates that success can be achieved even without knowledge of the rule, the rule is unnecessary, and the candidate test session should be eliminated. It is understood that "failed" here means that the lower bound check condition is met, i.e., the rule-blind agent cannot succeed.

[0110] Finally, from the candidate test sessions, the candidate test sessions whose upper bound check result is "pass" and whose lower bound check result is "fail" are selected as the officially retained rule binding test sessions.

[0111] This dual screening condition of "upper bound passed and lower bound not passed" ensures that agents who know the rules can succeed, while agents who do not know the rules will fail. This allows the test session to reliably distinguish between the two states of "rules acquired" and "rules not acquired", providing a solid and reliable foundation for subsequent evaluation.

[0112] In practice, for candidate test sessions that fail the above upper / lower bound checks, up to two rounds of refinement can be performed using the quality inspection failure trajectory before resubmitting for quality inspection. If the refinement still fails to pass both the upper and lower bound checks simultaneously, the candidate test session, along with its bound service rules, will be discarded to ensure a reliable one-to-one pairing between the retained rules and the test session.

[0113] This embodiment ensures the sufficiency of the rules for the task through upper bound checks and the necessity of the rules for the task through lower bound checks. In this way, this embodiment performs the two-boundary checks by majority voting of multiple independent judges, which not only avoids the systematic bias of a single model judgment, but also ensures the evaluation effectiveness of each test session through the joint determination of "upper bound passed and lower bound failed", thereby improving the reliability and interpretability of evaluation-related indicators.

[0114] After generating a test session bound to the service rules based on the service object in step S11, a learning session can be generated. For example, in one embodiment, the user information may include a persona and assistant domain; generating a learning session based on the user information of the service object and the test session may include: determining the task topic corresponding to the learning session based on the persona and assistant domain; instantiating the task topic into a learning dialogue based on the task topic and a pre-set dialogue format; performing signal overlap selection based on the test session to select learning dialogues from the instantiated learning dialogues where the agent's tool-level behavior overlaps with the agent's tool-level behavior in the test session; and selecting learning sessions other than those with overlapping tool-level behavior to fill in the gaps according to a predetermined number or proportion requirement of sessions, to form a learning session sequence.

[0115] Specifically, the generation of the learning session includes a skeleton phase and a filling phase. In the skeleton phase, based on the persona and assistant domain, a time-ordered pool of daily topics is sampled. Each topic contains only a domain label and a single line of topic description. This phase is not conditional on any target rule. In the filling phase, each task topic produced in the skeleton phase is instantiated into a complete learning session according to a predetermined format. This can include fuzzy user intent, specific task parameters, local reference objects, and the gold-labeled tool trajectory (correct behavior, what should be done) for performing the task. Each completed learning session can undergo shape verification and gold-labeled trajectory execution verification in a real environment. If both pass, it is retained. Subsequently, signal overlap selection is performed. This step takes the test session set as input (but is only used for action signal comparison) and can select approximately 2|T| sessions from the instantiated learning sessions (|T| is the total number of test sessions). Specifically, the selection logic can be: priority is given to including all learning sessions whose gold-labeled trajectories touch a certain service rule decision surface tool, that is, sessions where the tool-level behavior of the agent in the learning session and the tool-level behavior of the agent in the test session have "signal overlap" (called signal sessions). Then, sessions are randomly selected from the learning sessions that do not overlap with the tool-level behaviors of the service rules to fill the remaining slots. If the signaling session pool has exceeded the target length, a subset is randomly selected.

[0116] By using the above selection process, a significant proportion of hidden service rules have the opportunity to "naturally encounter their decision surfaces" in the learning session sequence, but no learning session is generated based on a rule. This preserves the rule-related decision opportunities in real service scenarios without revealing the hidden service rules. The selected sessions are output in the chronological order of their generation, forming an ordered learning session sequence.

[0117] This embodiment ensures the inherent consistency between session content and user profiles through a two-stage skeleton-filling generation process. It then uses signal overlap sequencing to ensure that a significant proportion of hidden service rules have a chance to "naturally encounter their decision surfaces" in the learning session sequence. Simultaneously, non-overlapping sessions are used to fill in gaps to meet the total number of learning sessions. Thus, this embodiment, through the "occasional but not disclosed" data construction principle, preserves the natural ecosystem of opportunities for agents to ask questions in real service scenarios while mechanistically avoiding the direct transcription and utilization of service rules. This allows the evaluation to focus on proactive acquisition capabilities rather than passive information retelling capabilities.

[0118] After generating the learning session, the agent to be evaluated is then evaluated using the learning session and the test session.

[0119] First, the learning session can be conducted between a simulated service object (user simulator) and the agent to be evaluated. During the learning session, questions asked by the agent to the user can be captured and adjudicated. For example, in one implementation, the question information asked by the agent to the simulated service object during the learning session includes the question content and intent description; after intercepting the question information asked by the agent to the simulated service object during the learning session, the method may further include: passing the question information to a preset router for bypass adjudication; the router adjudicating whether the question content is a long-term rule question based on the question content and intent description; if the adjudication is positive, passing the question content to a preset classifier to adjudicate whether the question content matches the service rule; in response to the router's positive judgment, sending the question information to the simulated service object... The service object adds a temporary instruction message, which requires the simulated service object to output a placeholder token. The instruction message is completely consistent under different decision results of the classifier, so that the simulated service object remains blind to the decision results of the classifier. When the simulated service object responds based on the temporary instruction message, the response message is intercepted. The placeholder token in the response message is replaced with the standard answer corresponding to the hit service rule when the decision result of the classifier is a hit. When the decision result of the classifier is a miss or an error, it is replaced with a fixed avoidance statement constant. The replaced response message is then returned to the agent.

[0120] Specifically, during the learning session, the agent's questions to the simulated service object can be sent through a unique user-facing channel. For example, this channel can take the form `send_to_user(output, reason)`, and its signature contains two parameters: `output` (question information) and `reason` (intent description). `output` is a unique agent string that enters the visible transcription, while `reason` remains hidden from the simulated service object and the classifier. All agent-facing messages are converged to this single channel, allowing the router to examine only those messages that are likely rule-based questions.

[0121] After intercepting the query information sent by the agent via send_to_user, the following bypass decision process is executed: First, the output (question content) and reason (intent description) of the question information are passed to a preset router. This router is a lightweight large language model frozen during the evaluation period, configured to determine whether the question is a "strict long-term rule question" based on the output and reason binary judgment of `is_strict_rule_question`. The router classifies direct and soft / indirect questions involving "future / default behavior" as positive; while recalling past rules, state updates, and referencing draft content are classified as negative. If the router classifies a question as positive, it copies the question segment `rule_question_span` verbatim from the output as the output; if it classifies a question as negative, the process terminates, and subsequent classifier and rule response hooks are not triggered, meaning the interaction between the agent and the simulated service object is not interfered with.

[0122] The question segment `rule_question_span` is submitted to a predefined classifier only when the router determines it is positive. This classifier can also be a lightweight, large language model frozen during the evaluation period, configured to match the question segment against the user's rule pool. The classifier's attention is specifically directed towards "topic-rule" matching rather than intent classification: the classifier only considers a hit if the rule-triggered context covers the question and the rule's preferred behavior directly answers the action / value / choice requested in the question; otherwise, it returns an empty string (miss). If the router determines it is positive but the classifier returns an empty string, or if the router determines it is positive but question segment extraction fails, it is treated as a "miss / error".

[0123] After the router determines the result is positive (regardless of the classifier's subsequent decision), a temporary role=user instruction message is appended to the simulated service object, which is the rule-answer hook. This instruction message requires the simulated service object to output a placeholder token once at its natural location.<RULE_ANSWER> Furthermore, it does not write any actual answers. Crucially, the instruction message is completely consistent across the classifier's three decision outcomes—hit, miss, and error. This means the simulated service object receives the same instruction and cannot distinguish which branch the current decision belongs to based on the instruction content, thus maintaining a "blind state" regarding the classifier's decision. This design structurally prevents the simulated service object from potentially revealing "which messages were treated as rule queries" due to knowledge of the decision outcome.

[0124] Finally, after the simulated service object replies based on the temporary instruction message, the reply message is intercepted, and token substitution is performed: if the classifier determines a hit, the placeholder token in the reply message is replaced.<RULE_ANSWER> The placeholder is replaced with the standard answer corresponding to the matched service rule. If the classifier determines a miss or error, the placeholder is replaced with a fixed avoidance statement constant (e.g., "I don't have a particular preference on this—you decide"). This avoidance statement constant is a fixed string throughout the entire evaluation process, rather than being temporarily created by the simulator, thus eliminating variance caused by the simulator "inventing preferences out of thin air" when a miss occurs. The response message after replacement is returned to the agent, allowing the agent to receive a natural language response corresponding to the ruling in the dialogue.

[0125] If the placeholder token is missing from the simulated service object's response (i.e., not output in the expected position)<RULE_ANSWER> The mechanism can initiate a retry mechanism up to two times, resending the same temporary command message each time to request the simulator to replenish the token; if the token is still missing after two retries, a replacement text is appended to the end of the reply as a backup delivery. The entire hook mechanism does not persist the temporary command message to the session track, ensuring the cleanliness of the persistent track.

[0126] This embodiment requires the simulated service object to output only a placeholder token by using consistent instructions in a form unrelated to the ruling, and then performs the replacement according to the ruling result afterward. This makes the simulator's behavior completely indistinguishable under different ruling branches, thereby fundamentally eliminating the possibility of the simulated service object leaking ruling information to the interaction process through its own behavioral differences. In this way, by replacing the improvised response when a miss occurs with a fixed avoidance language constant, this embodiment further eliminates the source of variance of the simulator's "inventing bias" when a miss occurs, making the response behavior during the learning phase statistically stable and controllable, and ensuring the purity and reproducibility of the evaluation process.

[0127] In one implementation, the execution of the learning session may include: running each learning session sequentially in chronological order, enabling the agent to interact with a simulated service object (simulator) to complete the current task, and granting the agent the authority to autonomously raise questions related to future tasks during the interaction; after each learning session, appending the cleaned session transcript to a cross-session context block; after all learning sessions have ended, freezing the accumulated cross-session context block, making it read-only and non-write-back, and keeping it fixed for the agent under test; next, running a test phase on the frozen cross-session context block (i.e., as the memory base of the agent under test), during which the agent is prohibited from raising new questions.

[0128] In one implementation, the execution of the learning session may include: converging all vocalizations of the agent toward the simulator to a single synthesis tool, which performs conversational transcription of visible text delivered to the simulator as an internal intent string for consumption by a bypass adjudication router, and the internal intent string is hidden from the simulator, the classifier, and the frozen context.

[0129] In one implementation, the execution of the learning session may include: when the agent issues a plain text round without tool calls in the learning session, triggering an offline protocol repair hook; the repair hook executes an inner retry loop of up to three times, each time temporarily injecting a correction prompt requiring the agent to resend via the unique user-facing channel; if the correction is successful, the original plain text message is replaced with the corrected message and the correction prompt is not persisted; if the message is still plain text after three retries, the plain text is rolled back and sent to the bypass arbitration router as a voice transmission to the user-facing channel.

[0130] After step S13, by simulating a service object and conducting the learning session with the agent, and intercepting the questions asked by the agent to the simulated service object during the learning session, step S14 can be executed to use the session trajectory of the learning session as the memory base of the agent, and conduct the test session with the agent when the simulated service object is offline.

[0131] After the test session with the agent concludes in step S14, a deterministic determination can be made regarding the agent's behavior, i.e., whether the agent acts in accordance with the service rules to obtain an evaluation result. For example, in one implementation, obtaining the evaluation result of the agent's proactive information acquisition capability based on the agent's questioning information to the simulated service object during the learning session and the execution information of the test session may include: determining, based on the execution information of the test session, whether the agent acts in accordance with the service rules bound to the test session; if the agent acts in accordance with the service rules bound to the test session, the agent is deemed to have passed the test; otherwise, the agent is deemed to have failed the test; based on the result of the determination and the agent's questioning information to the simulated service object during the learning session, the evaluation result of the agent's proactive information acquisition capability is obtained.

[0132] Specifically, the execution information of the test session includes the complete session trajectory of the agent during the testing phase, which records the tool calls and parameters of the agent at each step. The test session is bound to service rules, which carry tool-level behaviors and inspection types as judgment criteria.

[0133] When determining whether an agent acts in accordance with service rules, the conversation trajectory can be processed in batches according to the agent's assistant rounds. Parallel tool calls within the same assistant message are evaluated as a batch. Then, each batch is scanned sequentially from the beginning of the conversation, and the first batch containing a tool call matching the tool required by the service rules is identified. This batch is used as the judgment criterion. All subsequent batches are ignored; that is, the "first relevant decision" rather than the "final correction" is used as the judgment criterion. The reason for this is that the testing phase simulates an offline user state, and the user cannot immediately correct the agent after its first violation. Therefore, the first decision, rather than subsequent corrections, should be used as the evaluation benchmark.

[0134] After identifying the first relevant batch, further action comparisons can be performed on the tool-level behaviors within that batch based on the service rule's inspection type. The specific judgment method varies depending on the inspection type. For tool identity checks, the judgment conditions are relatively lenient, requiring only that there is a call to the required tool in the first matching batch; no further verification of the tool call parameters is involved. For object identifier checks, parameter enumeration value checks, and confirmation operation checks, the judgment conditions are more stringent. Each call to the same required tool in the first matching batch must match the specified comparison parameters, and a "correct call plus incorrect call" scattershot call method within the same round is explicitly rejected. That is, calls to multiple instances of the same tool within the same batch must not include calls that do not meet the parameter requirements. Regarding parameter value comparison methods, scalar parameters use exact equality judgment, list parameters use set equality judgment, and confirmation operation checks require further comparison of whether the target tool's security type field meets predetermined requirements.

[0135] If the tool-level behavior in the first relevant decision is confirmed to meet the service rules' tool-level behavior and inspection type requirements after the above comparison, the agent is determined to have passed the test session; otherwise, if the first relevant batch lacks the required tool call, or the call parameter value does not meet the gold standard requirements, or there are illegal instances in the shotgun call, the agent is determined to have failed the test session.

[0136] The pass / fail results of all test sessions are summarized to obtain the test session pass rate (TSAcc) of the agent, which is the ratio of the number of successful test sessions to the total number of test sessions.

[0137] Furthermore, based on the judgment result and the questioning information of the agent to the simulated service object during the learning session, a comprehensive evaluation result is obtained: the agent's proactive questioning behavior characteristics regarding service rules are extracted from the questioning information during the learning phase, and this is combined with the pass rate during the testing phase for analysis, forming a multi-dimensional evaluation result of the agent's proactive information acquisition capability. Specific decomposition diagnostic indicators will be described in detail in subsequent implementation methods.

[0138] This embodiment combines test results with question information from the learning phase, enabling the evaluation to encompass both objective pass rates at the behavioral level and behavioral profiles at the question level. This provides a data foundation for a multi-dimensional characterization of proactive acquisition capabilities. Furthermore, by treating "judgment results" and "question information" as two independent input dimensions, this embodiment allows the evaluation report to distinguish between two different failure modes: "failure due to not asking questions" and "asking questions but not using them correctly," thus improving the diagnosticability of the evaluation results.

[0139] To determine whether the agent acts in accordance with the service rules, this embodiment refines the specific process of action matching in order to achieve deterministic and procedural verification of the test results. For example, in one implementation, the execution information may include a session trajectory, and the service rules bound to the test session may include tool-level behaviors and inspection types, such as... Figure 3 As shown, determining whether the agent acts in accordance with the service rules bound to the test session based on the execution information of the test session may include: S201. Process the conversation trajectory in batches according to the assistant rounds of the intelligent agent; In this step, parallel tool calls within the same assistant message are evaluated as a batch. By processing in batches, multiple tools called in parallel are evaluated at the same decision point, avoiding misjudgments that can result from splitting a single decision into multiple independent decision points.

[0140] S202. Scan each batch sequentially from the beginning of the session trajectory to locate the first batch containing a tool call that matches the tool required in the tool-level behavior; In this step, this batch is used as the basis for judgment, and all subsequent batches are ignored. That is, the "first relevant decision" rather than the "final correction" is used as the basis for judgment. The reason is that the testing phase simulates the user's offline state, and the user cannot correct the agent's first violation in real time. Therefore, the first decision should be used as the evaluation benchmark rather than subsequent corrections.

[0141] S203. Based on the inspection type of the service rule, perform an action comparison on the tool-level behavior in the first relevant batch to determine whether the action in the tool-level behavior corresponding to the inspection type meets the gold standard requirements. In this step, the gold standard requirement refers to the correct action requirement in the tool-level behavior performed by the agent in the test session according to the service rules. The specific comparison method varies depending on the type of inspection: For tool identity checks, the criteria are that the requirement is met only if there is a call to the required tool in the first matching batch, without further verification of the tool call parameters. For object identification checks, parameter enumeration value checks, and confirmation operation checks, the judgment conditions are more stringent. It requires that each call to the same required tool in the first matching batch must match the specified comparison parameters, and explicitly rejects the "correct call plus incorrect call" shotgun call method in the same round. That is, calls that do not meet the parameter requirements must not be mixed in the multiple instances of the same tool in the same batch. Regarding the specific comparison method for parameter values, scalar parameters are judged by exact equality, list parameters are judged by set equality, and the confirmation operation check requires further comparison to see if the security type field of the target tool meets the predetermined requirements.

[0142] S204. If the tool-level behavior in the first relevant decision is confirmed to meet the tool-level behavior and inspection type requirements of the service rules after comparison, then the agent is determined to have passed the test session.

[0143] In this step, it means that the agent acts in accordance with the service rules bound to the test session in the test session; otherwise, if the first relevant batch lacks the required tool call, or the parameter value of the call does not meet the gold standard requirements, or there are illegal instances in the shotgun call, then the agent is determined to have failed the test session.

[0144] This embodiment, by procedurally dividing the evaluation into batches according to assistant rounds and using the "first relevant decision" rather than the "final correction" as the decision window, ensures that the evaluation results are calculated entirely from the trajectory by deterministic logic, avoiding scoring noise caused by different referee models or different runs of the same model. Thus, by making a decision at the first relevant decision point rather than waiting for subsequent corrections, this embodiment better reflects real-world deployment scenarios where users cannot immediately correct the agent when offline, thereby improving the predictive validity of the evaluation results in real-world application situations.

[0145] During the action comparison process, precise matching of parameter values ​​can be achieved. For example, in one implementation, the gold standard requirement may include a gold standard value for action parameters; the step of comparing the tool-level behaviors in the assistant round batch based on the inspection type to determine whether the actions in the tool-level behaviors corresponding to the inspection type meet the gold standard requirement may include: determining the target action parameters in the tool-level behaviors in the assistant round batch that correspond to the inspection type based on the inspection type; determining whether the target action parameter value is the same as the gold standard value for action parameters; if they are the same, determining that the target action meets the gold standard requirement.

[0146] Specifically, the gold standard requirement includes the gold standard value of the action parameter, which is the correct parameter value specified in the tool-level behavior of the service rule. During comparison, firstly, based on the check type, the target action parameter corresponding to the check type in the tool-level behavior of the first relevant batch is determined—that is, the specific parameter to be compared. For example, if the check type is param_id, the target action parameter is a *_id type parameter (such as contact_id, project_id, etc.); if the check type is param_enum, the target action parameter is an enumeration type parameter (such as cabin_class, priority_level, etc.); if the check type is confirm, the target action parameter is a security type field.

[0147] Then, the target action parameter value is deterministically compared with the action parameter gold standard value. For scalar parameters (such as a single id or a single enumeration value), exact equality is used; for list parameters (such as a set of multiple ids), set equality is used, which requires that the actual set of parameters called is exactly the same as the gold standard value set, and no additional additions or subtractions are allowed.

[0148] If the target action parameter value is the same as the gold standard value for action parameters, then the target action is determined to meet the gold standard requirements; if they are different (including incorrect parameter values, missing parameters, or "correct + incorrect" shotgun calls resulting in incomplete equality of sets), then it is determined not to meet the gold standard requirements. This deterministic comparison process is executed by the program without the need for LLM judges, ensuring the reproducibility and auditability of the scoring.

[0149] This embodiment represents the gold standard requirement as specific action parameter gold standard values ​​and defines the success condition as the exact equality of parameter values. This ensures that the judgment logic is completely consistent across different models and operating environments, eliminating the uncertainty caused by semantic approximation matching. Thus, by using exact equality for scalar parameters and set equality for list parameters, this embodiment refines the verification granularity of tool-level behavior to each parameter slot, improving the sensitivity of the evaluation to execution details and the diagnostic accuracy.

[0150] For different inspection types, corresponding comparisons can be performed on tool-level behaviors in the assistant round batch. For example, in one implementation, the inspection type may include tool identity checking, and the gold-label value of the action parameter may include a gold-label tool identifier. The action comparison of the tool-level behaviors in the assistant round batch based on the inspection type may include comparing whether the tool identifier called by the tool-level behaviors in the assistant round batch is the same as the gold-label tool identifier. For example, if the gold-label tool identifier is "send_email", then there must be a call to the send_email tool in this batch. This inspection type only verifies the correctness of the tool selection and does not involve its parameter values.

[0151] For example, in one implementation, the inspection type may include an object identifier inspection, the action parameter gold standard value may include a gold standard object identifier, and the action comparison of the tool-level behavior in the assistant round batch based on the inspection type may include: comparing whether the selected object identifier in the tool-level behavior in the assistant round batch is the same as the gold standard object identifier. For example, if the gold standard object identifier is "contact_12345", then all calls to the contact_select tool in this batch must set the contact_id parameter to "contact_12345".

[0152] For example, in one implementation, the inspection type may include parameter enumeration value inspection, and the action parameter gold standard value may include gold standard parameter enumeration values. The action comparison of the tool-level behaviors in the assistant round batch based on the inspection type may include comparing whether the parameter enumeration values ​​of the tools called in the tool-level behaviors of the assistant round batch are the same as the gold standard parameter enumeration values. For example, if the gold standard parameter enumeration value is "economy", then all calls to the flight_booking tool in this batch must have the cabin_class parameter set to "economy", and there must be no cases where some calls are set to "business".

[0153] For example, in one implementation, the check type may include a confirmation operation check, the action parameter gold standard value may include a confirmation tool identifier, and the action comparison of the tool-level behaviors in the assistant round batch based on the check type may include: comparing whether the tool identifier called before executing the change operation in the tool-level behaviors of the assistant round batch is the confirmation tool identifier. For example, if the confirmation tool identifier is "confirm_cancellation", then the agent must call the confirm_cancellation tool before calling the cancel_subscription tool, and the security type field of the confirmation call must be "confirmed".

[0154] The four comparison logics described above correspond to different tool-level decision types, with the selection determined by the check type field of the service rule. By defining specific comparison logic for each check type, coarse-grained verification of tool identity, fine-grained verification of object identifiers and parameter enumerations, and process sequence verification of confirmation operations can all be executed uniformly and precisely, and can be automated and executed accurately by machines. In this way, by being compatible with four different granularity check modes under the same decision framework, the evaluation system can cover the complete execution correctness path from "which tool to select" to "which object to select" to "which parameter value to select" and then to "whether the confirmation operation is performed first," improving the evaluation's adaptability to multi-form rules.

[0155] After obtaining the evaluation results of the agent's proactive information acquisition capability in step S15, in order to conduct multi-dimensional attribution analysis on the agent's proactive acquisition capability, the evaluation results can be jointly constructed by the test session pass rate and decomposed diagnostic indicators. For example, in one implementation, obtaining the evaluation result of the agent's proactive information acquisition capability based on the result of the determination and the questioning information of the agent to the simulated service object during the learning session may include: calculating the agent's test session pass rate based on the result of the determination, the test session pass rate being equal to the ratio of the number of test sessions passed by the agent to the total number of test sessions; calculating decomposition diagnostic indicators based on the questioning information of the agent to the simulated service object during the learning session; wherein the decomposition diagnostic indicators may include at least one of the following: number of rule questions, rule coverage, acquisition accuracy, and application rate, wherein the number of rule questions is the number of strict questions asked by the agent for the service rule in each learning session, the rule coverage is the proportion of service rules that are hit at least once in the question information to all service rules, the acquisition accuracy is the proportion of question information that hits the service rule to all question information, and the application rate is the agent test pass rate of the test sessions bound to the service rule in which questions were asked in the learning session; and obtaining the evaluation result of the agent's proactive information acquisition capability based on the test session pass rate and the decomposition diagnostic indicators.

[0156] The ruleAsk count refers to the number of strict rule questions issued by the agent in each learning session regarding the service rules. This metric measures the frequency with which the agent actively asks follow-up questions. During calculation, the number of questions that the router determines to be positive (is_strict_rule_question=True) is counted separately for each learning session, and an average can be calculated across sessions. If the ruleAsk count is zero in a learning session, it indicates that the agent in that session has no intention of actively seeking future reusable information.

[0157] The rule coverage rate (RuleCov) refers to the proportion of service rules that are matched at least once by the classifier in the query information out of all service rules. The formula is: RuleCov = Number of service rules matched at least once / Total number of service rules in the rule pool. This metric measures the breadth of rules actively acquired by the agent during the learning phase. For example, if there are 20 rules in the rule pool, and the agent's query matches 8 different rules, then RuleCov = 8 / 20 = 40%. If a rule is never matched by any query, even if it is successfully applied in a test session, it cannot be attributed to active acquisition.

[0158] The Acquisition Precision Rate (AcqPrec) refers to the proportion of strict rule queries mapped to a specific service rule in the hidden rule pool by the classifier out of all strict rule queries. Its calculation formula is: AcqPrec = Number of queries that hit the rule / Total number of queries correctly identified by the router. This metric measures whether the agent's queries accurately point to useful future information. A low AcqPrec indicates that although the agent queries frequently, a large number of these queries are "invalid queries" (not hitting any hidden rules), resulting in low acquisition efficiency.

[0159] The AppliedRate refers to the agent test pass rate of test sessions bound to service rules that are "learnable and confirmed by the classifier during the learning period". First, "learnable rules" are defined as rules whose actions are triggered by at least one gold-standard trajectory in a selected learning session. Then, rules confirmed by the classifier during the learning period are selected, and the pass rate of the test sessions bound to these rules is calculated. This metric is used to distinguish between "acquisition failure" and "application failure": if a rule has been successfully acquired (hit) during the learning period, but its bound test session still fails, the failure should be attributed to "insufficient application capability" rather than "failure to acquire".

[0160] Based on the above-mentioned decomposed diagnostic indicators, combined with the main test session pass rate (TSAcc), a multi-dimensional evaluation result can be formed: TSAcc reflects the agent's final task execution performance; RuleAsk and RuleCov reflect its willingness and breadth of proactive questioning; AcqPrec reflects the accuracy of its questions; and AppliedRate reflects its application ability under the premise of "already acquired" information. The differences between the indicators can help pinpoint the weak links in the agent's proactive information acquisition capabilities.

[0161] This embodiment calculates the pass rate by using four independently analyzable capability dimensions: the number of rule-based questions reflects the willingness to acquire information, rule coverage reflects the breadth of acquisition, acquisition precision reflects the quality of questions, and application rate reflects the effectiveness of execution and conversion after acquisition. In this way, by breaking down failures layer by layer into three attribution levels—"whether a question was asked, how accurate the question was, and whether the answer was applicable"—developers can accurately identify the specific weaknesses of the agent, providing data-driven guidance for targeted capability optimization.

[0162] To further quantify the difference in benefits and recyclability of actively acquiring this capability relative to known information, a diagnostic variant gradient mechanism can be introduced. For example, in one implementation, the method may further include: acquiring a learning variant session, which includes at least one of a default variant session, a soft-permission variant session, and a forced-question variant session, wherein the default variant session allows the agent to ask questions and is unrelated to the service rules; the soft-permission variant session does not disclose the service rules but prompts the agent to ask about the service rules at an appropriate time to obtain future task benefits; and the forced-question variant session requires the agent to ask about the service rules after the current task is completed; engaging in the learning variant session with the agent; and using the session trajectory of the learning variant session as the agent's memory base to engage in the... The process includes: testing a session; obtaining the pass rate of the test session after learning the variant session; obtaining a test variant session, wherein the test variant session is injected with the standard answer of the service rule as a prior user statement; conducting the test variant session with the agent; obtaining the pass rate of the test variant session; and obtaining the evaluation result of the agent's active information acquisition capability based on the test session pass rate and the decomposition diagnostic index, which may include: obtaining the evaluation result of the agent's active information acquisition capability based on the test session pass rate, the decomposition diagnostic index, the test pass rate of the test session after learning the variant session, and the test pass rate of the test variant session.

[0163] The default variant session can refer to a session in which only a single user-facing channel, send_to_user, is provided to the agent during the learning phase, without providing any prompts related to service rules. This allows the agent to determine whether and when to ask questions based entirely on its own capabilities, and is used to measure the baseline of the agent's spontaneous ATR behavior.

[0164] The soft license variant session (ATR variant) can refer to adding a soft license instruction to the agent during the learning phase that does not disclose any specific rules but prompts "you can ask the user about a long-term rule at an appropriate time" and includes a cost-benefit reminder, but does not disclose any specific rule content or the timing of the question, and is used to measure the degree of increase in the agent's willingness to ask questions after being granted generalization permission.

[0165] The forced questioning variant session (always_ask variant) refers to requiring the agent to ask a long-term rule question exactly after the current task is roughly completed during the learning phase. This fixes the timing of the questioning, ensuring that all tested models ask questions at the same point. This is used to separate the two types of failure: "not asking" and "asking the wrong question." If performance improves after forced questioning, it indicates that the original failure mainly came from "not asking." If failure still occurs after forced questioning, it indicates that the failure mainly comes from "asking the wrong question" or "using the wrong question."

[0166] For each learning variant session, interact with the agent separately, then use the session trajectory of the learning variant session as the agent's memory base (i.e., freeze the cross-session context), and then conduct a test session with the agent to obtain the test session pass rate under each learning variant.

[0167] In addition, it includes test variant sessions (oracle variants), which, without a learning phase, directly inject the standard answer of the bound service rule as a prior user statement (presented in "[Priorstatement]" format) into the agent in the test session. This gives the agent complete information about the rule in the test session, used to measure the upper bound of the test pass rate under "known preferences" conditions. After conducting the test variant session with the agent, the test session pass rate under that variant is obtained.

[0168] Ultimately, the evaluation results were obtained by comprehensively considering the following dimensions: the test pass rate of the default variant (baseline level), the test pass rate of the atr variant (soft licensing effect), the test pass rate of the always_ask variant (fixed timing effect), the test pass rate of the oracle variant (upper bound level), and the decomposed diagnostic metrics (RuleAsk, RuleCov, AcqPrec, AppliedRate).

[0169] Specifically, the pass rate gap between the oracle variant and the default variant (gap = oracle - default) is calculated, representing "how much actionable benefit can be obtained if the relevant preferences are known"; the improvement of each non-oracle variant relative to the default variant is calculated, and divided by the gap to obtain the "recovery rate", representing "how much of the information the agent recovers from its own questions"; then, combined with decomposition diagnostic indicators, the specific reasons for insufficient recovery are analyzed (whether not enough questions were asked, whether the questions were not accurate enough, or whether the answers were used incorrectly). Through the above multidimensional analysis, the agent's proactive information acquisition ability is attributed to three distinguishable links: "willingness (whether to ask)," "accuracy (how accurate the questions are)," and "applicability (how correctly the information is used)," thereby achieving fine-grained evaluation and attribution of ATR capabilities.

[0170] This embodiment forms a complete diagnostic gradient by applying variant configurations with different question instruction intensities to the same scenario data. This is achieved by forcing the question variant to separate failures of "not asking" and "asking inaccurately," using the soft-permission variant to separate the difference between "insufficient initiative" and "instruction compliance," and using the Oracle variant to provide an application upper bound. In this way, this embodiment quantifies the gap between the pass rate of the default variant and the Oracle upper bound as "the executable gains lost due to insufficient proactive acquisition," and quantifies how much of this gap can be filled by different intensities of prompt intervention using the return rate of each intermediate variant. This upgrades the evaluation from a simple "score" to an attributable and diagnostic capability profile analysis, providing a systematic measurement framework for understanding and improving the agent's proactive acquisition capabilities.

[0171] The evaluation method for intelligent agents provided by the embodiments of the present invention will be described in detail below through specific examples.

[0172] like Figure 4 As shown, the embodiment of the present invention provides an evaluation method for intelligent agents, the method including: S501. Obtain the service rules of the service object based on the user information of the service object; S502. Decompose the service rule into elements to determine the tool-level behavior and counterfactual default corresponding to the service rule; S503. Based on the tool-level behaviors and counterfactual defaults corresponding to the service rules, construct a test session. The test session includes user instructions, gold standard requirements, and interference items. The gold standard requirements are the correct action requirements in the tool-level behaviors that the agent performs when responding to the user instructions according to the service rules. The interference items are incorrect action results that conform to the counterfactual defaults. In the test session, the agent can act according to the service rules when the service rules are known. S504. Based on the user information of the service object and the test session, a learning session is generated. The learning session does not disclose the service rules, but the tool-level behaviors of the agents in some of the learning sessions overlap with the tool-level behaviors of the agents in the test session. S505. The learning session is conducted between the simulated service object and the intelligent agent, and the questioning information of the intelligent agent to the simulated service object during the learning session is intercepted, wherein the questioning information of the intelligent agent to the simulated service object during the learning session includes the question content and intent description; S506. The query information is handed over to a preset router for bypass decision-making. The router decides whether the query content is a long-term rule problem based on the query content and intent description. If the decision is positive, the query content is handed over to a preset classifier to decide whether the query content hits the service rule. S507. In response to the router's positive decision, a temporary instruction message is appended to the simulated service object. The instruction message requires the simulated service object to output a placeholder token, and the instruction message is completely consistent under different decision results of the classifier, so that the simulated service object remains blind to the classifier's decision result. S508. When the simulated service object replies based on the temporary instruction message, the reply message is intercepted, and the placeholder token in the reply message is replaced with the standard answer corresponding to the service rule when the classifier's decision result is a hit, and replaced with a fixed avoidance language constant when the classifier's decision result is a miss or an error, and the replaced reply message is returned to the agent. S509. Using the session trajectory of the learning session as the memory base of the agent, and conducting the test session with the agent when the simulated service object is offline; S510. Based on the execution information of the test session, determine whether the agent acts in accordance with the service rules bound to the test session in the test session; S511. If the agent acts in accordance with the service rules bound to the test session in the test session, the agent is determined to have passed the test; otherwise, the agent is determined to have failed the test. S512. Based on the result of the determination and the questioning information of the agent to the simulated service object during the learning session, obtain the evaluation result of the agent's active information acquisition capability.

[0173] Secondly, embodiments of the present invention also provide an evaluation device for intelligent agents, which can effectively evaluate the active information acquisition capabilities of intelligent agents.

[0174] like Figure 5 As shown, the intelligent agent evaluation device provided in the embodiments of the present invention may include: A test generation unit 21 is used to generate a test session bound to the service rules based on the service rules of the service object. In the test session, the agent can act according to the service rules when it knows the service rules. The learning unit 22 is used to generate learning sessions based on the user information of the service object and the test session. The learning sessions do not disclose the service rules, but the tool-level behaviors of the agents in some of the learning sessions overlap with the tool-level behaviors of the agents in the test session. The learning unit 23 is used to conduct the learning session between the simulated service object and the agent to be evaluated, and to intercept the question information of the agent to the simulated service object during the learning session; The test unit 24 is used to use the session trajectory of the learning session as the memory base of the agent, and to conduct the test session with the agent when the simulated service object is offline; The result acquisition unit 25 is used to acquire the evaluation result of the agent's active information acquisition capability based on the agent's questioning information to the simulated service object during the learning session and the execution information of the test session.

[0175] The agent evaluation device provided in the embodiments of the present invention can generate test sessions bound to service rules based on service rules, and generate learning sessions based on user information and test sessions that do not disclose service rules but have overlapping tool-level behaviors. It conducts learning sessions by simulating service objects and intercepts the agent's questions to the simulated service objects. Using the session trajectory of the learning sessions as a memory base, it conducts test sessions while the simulated service objects are offline. Based on the intercepted question information during the learning sessions and the execution information of the test sessions, it obtains the evaluation results of the agent's proactive information acquisition ability. Thus, because the learning sessions do not disclose service rules but some of their tool-level behaviors overlap with those in the test sessions, and offline testing is conducted using the learning session trajectory as a memory base, and because the agent can act according to the rules in the test sessions when the service rules are known, it can directly reflect the agent's proactive exploration of user rules when information is not pre-disclosed, based on the agent's questions to the simulated service objects in the learning sessions and the execution information of subsequent test sessions. This avoids the shortcomings of existing evaluations that can only assess the agent's retrieval of known information or application of disclosed information but cannot reflect proactive acquisition capabilities, thereby effectively evaluating the agent's proactive information acquisition ability.

[0176] In one embodiment, the apparatus further includes: a rule acquisition unit, configured to acquire the service rules of the service object based on the user information of the service object before generating a test session bound to the service rules based on the service rules of the service object.

[0177] In one implementation, the rule acquisition unit is specifically used to: provide the user information of the service object to a preset service rule generation model, and parse the user information and generate service rules for the service object through the service rule generation model.

[0178] In one implementation, the service rule carries at least the following information: standard answer, counterfactual default, tool-level behavior, and the inspection type corresponding to the tool-level behavior.

[0179] In one implementation, the inspection type includes at least one of the following: tool identity inspection, object identifier inspection, parameter enumeration value inspection, and confirmation operation inspection, wherein the tool identity inspection is used to verify whether the correct tool has been called, the object identifier inspection is used to verify whether the correct referenced object has been selected, the parameter enumeration value inspection is used to verify whether the correct parameter enumeration value has been selected, and the confirmation operation inspection is used to verify whether the confirmation tool has been called before the change operation is performed.

[0180] In one implementation, the service rule carries at least the following information: a standard answer and a tool-level behavior; the rule acquisition unit is specifically used to: acquire candidate service rules for the service object based on the user information of the service object; perform quality inspection on the acquired candidate service rules, the quality inspection including counterfactual default check and binding integrity check, wherein the counterfactual default check is used to determine whether an agent unaware of the candidate service rule will adopt the standard answer action of the candidate service rule, and if not, the counterfactual default check passes; the binding integrity check is used to determine whether the candidate service rule is mapped to the tool-level behavior, and if so, the binding integrity check passes; and the candidate service rule that passes both the counterfactual default check and the binding integrity check is taken as the service rule of the service object.

[0181] In one implementation, the test generation unit 21 is specifically used to: decompose the service rule into elements to determine the session elements corresponding to the service rule, wherein the session elements include at least an interaction scenario, tool-level behavior, and counterfactual default; and construct a test session based on the tool-level behavior and counterfactual default corresponding to the service rule, wherein the test session includes user instructions, gold standard requirements, and interference items, wherein the gold standard requirements are the correct action requirements in the tool-level behavior executed by the agent in response to the user instruction according to the service rule, and the interference items are the incorrect action results that conform to the counterfactual default.

[0182] In one implementation, the test generation unit 21 is specifically used for: constructing candidate test sessions based on the service rules of the service object; performing upper bound checks and lower bound checks on the candidate test sessions, wherein the upper bound check is used to inject the service rules into a preset rule-aware agent and conduct the candidate test sessions with the preset rule-aware agent; if a predetermined proportion of the actions in the tool-level behaviors executed by the rule-aware agent meet the gold standard requirements, the upper bound check is deemed to have passed; the lower bound check is used to conduct the test sessions with a preset rule-blind agent that does not know the service rules; if a predetermined proportion of the actions in the tool-level behaviors executed by the rule-blind agent do not meet the gold standard requirements, the lower bound check is deemed to have failed; and selecting candidate test sessions from the candidate test sessions whose upper bound check results are passed and whose lower bound check results are failed, as the test sessions.

[0183] In one implementation, the user information includes a user profile and an assistant domain; the learning generation unit 22 is specifically used to: determine the task topic corresponding to the learning session based on the user profile and assistant domain; instantiate the task topic into a learning dialogue based on the task topic and a pre-set dialogue format; perform signal overlap selection based on the test session to select learning dialogues from the instantiated learning dialogues where the agent's tool-level behavior overlaps with the agent's tool-level behavior in the test session; and select learning dialogues other than the learning dialogues with overlapping tool-level behaviors to fill in the gaps according to a predetermined number or proportion requirement of dialogues, so as to form a learning dialogue sequence.

[0184] In one implementation, the questioning information from the agent to the simulated service object during the learning session includes the question content and intent description; the learning unit 23 is specifically used to: conduct the learning session with the agent through the simulated service object, and intercept the questioning information from the agent to the simulated service object during the learning session; pass the questioning information to a preset router for bypass adjudication, the router adjudicating whether the question content is a long-term rule question based on the question content and intent description, if the adjudication is positive, the question content is passed to a preset classifier to adjudicate whether the question content matches the service rule; in response to the router's adjudication... If the decision is positive, a temporary instruction message is appended to the simulated service object. The instruction message requires the simulated service object to output a placeholder token, and the instruction message is completely consistent under different decision results of the classifier, so that the simulated service object remains blind to the classifier's decision result. When the simulated service object responds based on the temporary instruction message, the response message is intercepted. The placeholder token in the response message is replaced with the standard answer corresponding to the hit service rule when the classifier's decision result is a hit, and replaced with a fixed avoidance statement constant when the classifier's decision result is a miss or an error. The replaced response message is then returned to the agent.

[0185] In one embodiment, the result acquisition unit 25 includes: a determination module, configured to determine whether the agent acts in accordance with the service rules bound to the test session in the test session based on the execution information of the test session; a judgment module, configured to determine that the agent passes the test if the agent acts in accordance with the service rules bound to the test session in the test session, otherwise determine that the agent fails the test; and an acquisition module, configured to acquire an evaluation result of the agent's proactive information acquisition capability based on the judgment result and the questioning information of the agent to the simulated service object during the learning session.

[0186] In one implementation, the execution information includes a session trajectory; the service rules bound to the test session include tool-level behaviors and inspection types; the determining module is specifically used to: divide the session trajectory into batches according to the assistant rounds of the agent; scan the batched session trajectories to obtain the first batch of assistant rounds containing the tool-level behaviors; based on the inspection types, perform action comparison on the tool-level behaviors in the batch of assistant rounds to determine whether the actions in the tool-level behaviors corresponding to the inspection types meet the gold standard requirements, the gold standard requirements being the correct action requirements in the tool-level behaviors executed by the agent in the test dialogue according to the service rules; if the actions in the tool-level behaviors corresponding to the inspection types meet the gold standard requirements, then the agent is determined to have passed the test, indicating that the agent acted in the test session according to the service rules bound to the test session.

[0187] In one implementation, the gold standard requirement includes a gold standard value for action parameters; the determining module is specifically used to: determine, based on the inspection type, the target action parameter corresponding to the inspection type in the tool-level behavior of the assistant round batch; determine whether the target action parameter value is the same as the gold standard value for action parameters, and if they are the same, determine that the target action meets the gold standard requirement.

[0188] In one implementation, the determining module is specifically used for: the check type including tool identity check, the action parameter gold standard value including gold standard tool identifier, comparing whether the tool identifier called in the tool-level behavior in the assistant round batch is the same as the gold standard tool identifier; and / or, the check type including object identifier check, the action parameter gold standard value including gold standard object identifier, comparing whether the selected object identifier in the tool-level behavior in the assistant round batch is the same as the gold standard object identifier; and / or, the check type including parameter enumeration value check, the action parameter gold standard value including gold standard parameter enumeration value; comparing whether the parameter enumeration value of the tool called in the tool-level behavior in the assistant round batch is the same as the gold standard parameter enumeration value; and / or, the check type including confirmation operation check, the action parameter gold standard value including confirmation tool identifier, comparing whether the tool identifier called before performing the change operation in the tool-level behavior in the assistant round batch is the confirmation tool identifier.

[0189] In one implementation, the acquisition module is specifically used to: calculate the test session pass rate of the agent based on the determination result, wherein the test session pass rate is equal to the ratio of the number of test sessions passed by the agent to the total number of test sessions; calculate decomposition diagnostic indicators based on the questioning information of the agent to the simulated service object during the learning session; wherein the decomposition diagnostic indicators include at least one item: rule question count, rule coverage, acquisition accuracy, and application rate, wherein the rule question count is the number of strict questions asked by the agent for the service rule in each learning session, the rule coverage is the proportion of service rules that are hit at least once in the question information to all service rules, the acquisition accuracy is the proportion of question information that hits the service rule to all question information, and the application rate is the agent test pass rate of the test session bound to the service rule that was asked in the learning session; and obtain the evaluation result of the agent's proactive information acquisition capability based on the test session pass rate and the decomposition diagnostic indicators.

[0190] In one embodiment, the apparatus further includes a variant evaluation unit, the variant evaluation unit being configured to: acquire a learning variant session, the learning variant session including at least one of a default variant session, a soft-licensing variant session, and a forced-question variant session, wherein the default variant session allows the agent to ask questions and is unrelated to the service rules, the soft-licensing variant session does not disclose the service rules but prompts the agent that asking the service rules at an appropriate time can obtain future task benefits, and the forced-question variant requires the agent to ask service rule questions after the current task is completed; conduct the learning variant session with the agent; use the session trajectory of the learning variant session as the agent's memory base, conduct the test session with the agent; acquire the test pass rate of the test session after the learning variant session; and acquire a test variant session, the test variant session injecting the standard answer of the service rules as a priori user statement; conduct the test variant session with the agent; and acquire the test pass rate of the test variant session. The acquisition module is specifically used to: obtain the evaluation results of the agent's active information acquisition capability based on the test session pass rate, the decomposition diagnostic indicators, the test pass rate of the test session after the learning variant session, and the test pass rate of the test variant session.

[0191] Thirdly, embodiments of the present invention also provide an electronic device capable of effectively evaluating the active information acquisition capabilities of an intelligent agent.

[0192] like Figure 6As shown, the electronic device provided in the embodiments of the present invention may include: a processor 71 and a memory 72, wherein the processor 71 and the memory 72 are electrically connected; the memory 72 is used to store executable program code; the processor 71 runs a program corresponding to the executable program code by reading the executable program code stored in the memory 72, so as to implement any of the intelligent agent evaluation methods provided in the foregoing embodiments.

[0193] The specific execution process of the above steps by the processor 71, as well as the steps further executed by the processor 71 by running executable program code, can be found in the description of the foregoing embodiments, and will not be repeated here.

[0194] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement any of the intelligent agent evaluation methods provided in the foregoing embodiments, and thus can also achieve the corresponding technical effects. This has been described in detail above and will not be repeated here.

[0195] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0196] The various embodiments in this specification are described in a related manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0197] In particular, the device embodiment is basically similar to the method embodiment, so the description is relatively simple. For relevant details, please refer to the description of the method embodiment.

[0198] For ease of description, the above apparatus is described by dividing it into various functional units / modules. Of course, in implementing this invention, the functions of each unit / module can be implemented in one or more software and / or hardware.

[0199] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0200] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for evaluating an intelligent agent, characterized in that, include: Based on the service rules of the service object, a test session is generated that is bound to the service rules. In the test session, the agent can act in accordance with the service rules when it knows the service rules. Based on the user information of the service object and the test session, a learning session is generated. The learning session does not disclose the service rules, but the tool-level behaviors of the agents in some of the learning sessions overlap with the tool-level behaviors of the agents in the test session. The learning session is conducted between a simulated service object and the agent to be evaluated, and the questioning information of the agent to the simulated service object during the learning session is intercepted. The session trajectory of the learning session is used as the memory base of the agent, and the test session is conducted with the agent when the simulated service object is offline; Based on the agent's questioning information to the simulated service object during the learning session and the execution information of the test session, the evaluation results of the agent's proactive information acquisition capability are obtained.

2. The evaluation method according to claim 1, characterized in that, Before generating a test session bound to the service rule based on the service object, the method further includes: Based on the user information of the service object, obtain the service rules of the service object.

3. The evaluation method according to claim 2, characterized in that, The service rules for obtaining the service object based on the user information of the service object include: The user information of the service object is provided to the preset service rule generation model, and the service rule generation model parses the user information and generates service rules for the service object.

4. The evaluation method according to claim 1, characterized in that, During the learning session, the information the agent asks the simulated service object includes the content of the question and an explanation of its intent. After intercepting the agent's questioning information to the simulated service object during the learning session, the method further includes: The query information is passed to a preset router for bypass decision-making. The router decides whether the query content is a long-term rule issue based on the query content and intent description. If the decision is positive, the query content is passed to a preset classifier to decide whether the query content matches the service rule. In response to the router's positive decision, a temporary instruction message is appended to the simulated service object. The instruction message requires the simulated service object to output a placeholder token, and the instruction message is completely consistent under different decision results of the classifier, so that the simulated service object remains in a blind state to the classifier's decision result. When the simulated service object replies based on the temporary instruction message, the reply message is intercepted. The placeholder token in the reply message is replaced with the standard answer corresponding to the service rule when the classifier's decision result is a hit. When the classifier's decision result is a miss or an error, it is replaced with a fixed avoidance statement constant. The replaced reply message is then returned to the agent.

5. The evaluation method according to claim 1, characterized in that, The evaluation results of the agent's proactive information acquisition capability, based on the agent's questioning information to the simulated service object during the learning session and the execution information of the test session, include: Based on the execution information of the test session, determine whether the agent acts in accordance with the service rules bound to the test session during the test session; If the agent acts in accordance with the service rules bound to the test session during the test session, the agent is deemed to have passed the test; otherwise, the agent is deemed to have failed the test. Based on the judgment result and the questioning information of the agent to the simulated service object during the learning session, the evaluation result of the agent's proactive information acquisition capability is obtained.

6. The evaluation method according to claim 5, characterized in that, The execution information includes the session trajectory; The service rules bound to the test session include tool-level behaviors and inspection types; Determining whether the agent acts in accordance with the service rules bound to the test session based on the execution information of the test session includes: The conversation trajectory is divided into batches according to the assistant rounds of the intelligent agent; Scan the conversation trajectories in batches to obtain the first assistant round batch containing the tool-level behavior; Based on the inspection type, the tool-level behaviors in the assistant round batch are compared to determine whether the actions in the tool-level behaviors that correspond to the inspection type meet the gold standard requirements. The gold standard requirements are the correct action requirements in the tool-level behaviors that the agent performs when acting in accordance with the service rules in the test dialogue. If the action corresponding to the inspection type in the tool-level behavior meets the gold standard requirements, the agent is determined to have passed the test, indicating that the agent acts in accordance with the service rules bound to the test session in the test session.

7. The evaluation method according to claim 5, characterized in that, The evaluation result of the agent's proactive information acquisition capability, based on the determination result and the agent's questioning information to the simulated service object during the learning session, includes: Based on the determination result, the test session pass rate of the agent is calculated, and the test session pass rate is equal to the ratio of the number of test sessions that the agent passed the test to the total number of test sessions; Based on the questions asked by the agent to the simulated service object during the learning session, decomposed diagnostic indicators are calculated; wherein, the decomposed diagnostic indicators include at least one: number of rule questions, rule coverage, acquisition accuracy, and application rate, wherein the number of rule questions is the number of strict questions asked by the agent for the service rule in each learning session, the rule coverage is the proportion of service rules that are hit at least once in the question information to all service rules, the acquisition accuracy is the proportion of question information that hits the service rule to all question information, and the application rate is the agent test pass rate of the test session bound to the service rule that was asked in the learning session; Based on the test session pass rate and the decomposition diagnostic indicators, the evaluation results of the agent's proactive information acquisition capability are obtained.

8. An evaluation device for an intelligent agent, characterized in that, include: A test unit is generated to generate a test session bound to the service rules of the service object. In the test session, the agent can act in accordance with the service rules when the service rules are known. A learning unit is generated to generate learning sessions based on the user information of the service object and the test session. None of the learning sessions disclose the service rules, but the tool-level behaviors of the agents in some of the learning sessions overlap with the tool-level behaviors of the agents in the test session. A learning unit is used to conduct the learning session between a simulated service object and the agent to be evaluated, and to intercept the questioning information of the agent to the simulated service object during the learning session; The testing unit is used to use the session trajectory of the learning session as the memory base of the agent, and to conduct the test session with the agent when the simulated service object is offline; The result acquisition unit is used to acquire the evaluation results of the agent's proactive information acquisition capability based on the agent's questioning information to the simulated service object during the learning session and the execution information of the test session.

9. An electronic device, characterized in that, The electronic device includes: a processor and a memory, wherein the processor is electrically connected to the memory; the memory is used to store executable program code; the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, so as to implement the evaluation method of the intelligent agent according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the evaluation method for an intelligent agent according to any one of claims 1 to 7.

Images