An agent evaluation method and system

Abstract

The embodiment of the application discloses an agent evaluation method and system. The method is applied to a controller, the controller is in communication connection with a browser, and a target webpage representing an agent of a webpage end is opened on the browser. The method comprises the following steps: obtaining a target question used for agent evaluation and a target answer configured for the target question; sending the target question to the browser, so that the browser sends the target question to the agent through the target webpage, and receives an output answer output by the agent for the target question through the target webpage, and returns the output answer; and receiving the output answer, so as to evaluate the question and answer performance of the agent by comparing the output answer with the target answer. The technical scheme of the embodiment of the application can realize agent evaluation, and in particular, can realize evaluation of the question and answer performance of the agent.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of electrical data digital processing technology, and in particular to a method and system for evaluating intelligent agents. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) are widely used, making the scientific evaluation of their performance particularly important. Currently, model evaluation is mainly conducted through application programming interfaces (APIs).

[0003] Furthermore, in practical applications of large language models, domain adaptation can be achieved through model fine-tuning, retrieval-augmented generation (RAG), and prompt engineering, thereby guiding them to generate more domain-specific output. These deeply customized and encapsulated large language models can be considered intelligent agents driven by large language models.

[0004] Based on this, it should be noted that, compared to large language models with open APIs, most intelligent agents are provided in the form of Software as a Service (SaaS) and do not have open APIs. This makes it impossible to evaluate intelligent agents using the above methods, which urgently needs to be addressed. Summary of the Invention

[0005] This invention provides a method and system for evaluating intelligent agents, which solves the problem of the inability to evaluate intelligent agents.

[0006] According to one aspect of the present invention, an agent evaluation method is provided. This method is applied to a controller, which is communicatively connected to a browser. The browser displays a target webpage representing a webpage-based agent. The method may include:

[0007] Obtain the target question for agent evaluation and the target answer configured for the target question;

[0008] The target question is sent to the browser, so that the browser sends the target question to the agent through the target webpage, and receives the output answer of the agent in response to the target question through the target webpage, and returns the output answer;

[0009] Receive the output answer to evaluate the agent's question-answering performance by comparing the output answer with the target answer.

[0010] According to another aspect of the present invention, an agent evaluation method is provided. This method is applied in a browser, the browser being communicatively connected to a controller, and a target webpage representing a webpage-based agent is opened on the browser. The method may include:

[0011] Receive the target question sent by the controller for agent evaluation;

[0012] The target question is sent to the intelligent agent via the target webpage, and the intelligent agent's output answer to the target question is received via the target webpage.

[0013] The output answer is returned to the controller, which then evaluates the agent's question-answering performance by comparing the output answer with the target answer configured for the target question.

[0014] According to another aspect of the present invention, an intelligent agent evaluation system is provided, which may include: a controller and a browser communicatively connected to the controller, wherein a target webpage representing a web-based intelligent agent is opened on the browser; wherein,

[0015] The controller can be used to obtain the target question for agent evaluation and the target answer configured for the target question, and send the target question to the browser;

[0016] A browser can be used to receive a target question, send the target question to an agent via a target webpage, receive the agent's output answer in response to the target question via the target webpage, and return the output answer to the controller.

[0017] The controller can also be used to receive output answers in order to evaluate the agent's question-answering performance by comparing the output answers with the target answers.

[0018] According to another aspect of the present invention, an agent evaluation device is provided, the device being configured in a controller, the controller being communicatively connected to a browser, the browser having a target webpage representing a webpage-based agent open on the browser, the device comprising:

[0019] The target answer acquisition module can be used to acquire the target question for agent evaluation and the target answer configured for the target question.

[0020] The target question sending module can be used to send the target question to the browser, so that the browser can send the target question to the agent through the target webpage, and receive the output answer of the agent in response to the target question through the target webpage, and return the output answer;

[0021] The agent evaluation module can be used to receive output answers and evaluate the agent's question-answering performance by comparing the output answers with the target answers.

[0022] According to another aspect of the present invention, an agent evaluation device is provided, the device being configured in a browser, the browser being communicatively connected to a controller, and a target webpage representing a webpage-based agent being opened on the browser; the device may include:

[0023] The target question receiving module can be used to receive target questions sent by the controller for agent evaluation;

[0024] The output answer receiving module can be used to send a target question to an agent via a target webpage, and receive the output answer from the agent in response to the target question via the target webpage.

[0025] The output answer return module can be used to return the output answer to the controller, so that the controller can evaluate the agent's question-answering performance by comparing the output answer with the target answer configured for the target question.

[0026] According to another aspect of the present invention, a control terminal is provided, which may include:

[0027] At least one processor; and

[0028] A memory that is communicatively connected to at least one processor; wherein,

[0029] The memory stores a computer program that can be executed by at least one processor to implement the agent evaluation method provided in any embodiment of the present invention when executed by at least one processor.

[0030] According to another aspect of the present invention, a user terminal is provided, which may include:

[0031] At least one processor; and

[0032] A memory that is communicatively connected to at least one processor; wherein,

[0033] The memory stores a computer program that can be executed by at least one processor to implement the agent evaluation method provided in any embodiment of the present invention when executed by at least one processor.

[0034] According to another aspect of the present invention, a computer-readable storage medium is provided having computer instructions stored thereon for causing a processor to execute and implement the agent evaluation method provided in any embodiment of the present invention.

[0035] According to another aspect of the present invention, a computer program product is provided, on which a computer program is stored, which, when executed by a processor, implements the agent evaluation method provided in any embodiment of the present invention.

[0036] The technical solution of this invention involves a controller acquiring a target question for agent evaluation and a target answer configured for that question. The controller then sends the target question to a browser. The browser then sends the target question to the agent via the target webpage representing the agent, and receives the agent's output answer. The output answer is returned to the controller, which then compares the received output answer with the target answer to evaluate the agent's question-and-answer performance. This automated browser control facilitates the simulation of real user-agent interaction using the target webpage, enabling automated evaluation of the agent's question-and-answer performance. This is particularly suitable for automated evaluation of agents with non-publicly accessible APIs, customized through model fine-tuning, RAG (Rapid Application Generation), and prompt word engineering.

[0037] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 This is an example architecture diagram of an intelligent agent evaluation system provided according to an embodiment of the present invention;

[0040] Figure 2 This is a flowchart of an agent evaluation method provided according to an embodiment of the present invention;

[0041] Figure 3 This is a flowchart of another agent evaluation method provided according to an embodiment of the present invention;

[0042] Figure 4 This is a flowchart of another agent evaluation method provided according to an embodiment of the present invention;

[0043] Figure 5 This is a flowchart illustrating an example of agent response (i.e., output answer) integrity monitoring in another agent evaluation method provided according to an embodiment of the present invention;

[0044] Figure 6 This is a structural block diagram of an intelligent agent evaluation system provided according to an embodiment of the present invention;

[0045] Figure 7 This is a structural block diagram of an intelligent agent evaluation device provided according to an embodiment of the present invention;

[0046] Figure 8 This is a structural block diagram of another intelligent agent evaluation device provided according to an embodiment of the present invention;

[0047] Figure 9 This is a schematic diagram of the structure of the control terminal or user terminal that implements the intelligent agent evaluation method of the present invention. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The same applies to "target," "original," etc., and will not be repeated here. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in the technical solution of this invention all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to maintain user personal information security and network security.

[0051] Before introducing the embodiments of the present invention, the application scenarios of the embodiments of the present invention will be described by way of example. For example, see [link to example]. Figure 1This invention relates to an evaluation control terminal (hereinafter referred to as the control terminal) and a user terminal with a browser deployed on it. The control terminal can communicate with the browser via a remote debugging protocol, and the browser has a target webpage (Web) that can characterize a web-based intelligent agent. Based on this, the control terminal can send a target question to the browser, which can then send the target question to the intelligent agent via the target webpage, extract the output answer from the intelligent agent on the target webpage, and then obtain and store an evaluation result of the intelligent agent's question-answering performance based on the target question, the output answer, and the target answer configured for the target question. This will be described in detail below.

[0052] Figure 2 This is a flowchart of an agent evaluation method provided in an embodiment of the present invention. This embodiment is applicable to the automatic evaluation of agents, especially to the automatic evaluation of agents constructed based on model fine-tuning, RAG, and cue word engineering. The method can be executed by the agent evaluation device provided in this embodiment of the present invention. This device can be implemented in software and / or hardware, and can be configured on a controller, which can be integrated into a control terminal.

[0053] See Figure 2 The method of this invention specifically includes the following steps:

[0054] S110. Obtain the target question for evaluating the agent and the target answer configured for the target question.

[0055] Here, the target question can be understood as a question used to evaluate the question-answering performance of the intelligent agent. Based on this, and considering the application scenarios that may be involved in the embodiments of the present invention, optionally, the question can be obtained from a predefined evaluation dataset; optionally, the number of questions can be one or more, and if there are multiple questions, they can be processed sequentially; further optionally, the question can be a subjective question or an objective question; and so on. All of the above can be set according to the actual situation, and no specific limitation is made here.

[0056] The target answer can be understood as a pre-configured answer for the corresponding target question, especially a pre-configured standard answer, which is used to compare with the output answer of the agent for the target question.

[0057] The controller acquires the target question and its corresponding target answer. In this step, the predefined target question and answer ensure the repeatability and standardization of the agent's evaluation process, thereby avoiding subjective bias and providing a reliable basis for subsequent evaluation of the agent's question-and-answer performance.

[0058] S120. Send the target question to the browser, so that the browser sends the target question to the agent through the target webpage, receives the output answer of the agent for the target question through the target webpage, and returns the output answer.

[0059] In this context, the browser can be understood as a web browser deployed on the user's terminal, capable of loading and running the target webpage. The target webpage can be understood as the user interface representing the webpage-based intelligent agent. The webpage-based intelligent agent (i.e., the intelligent agent on the webpage) can be understood as an intelligent agent driven by a large language model and encapsulated as a webpage application; more specifically, it can be understood as an intelligent agent built on the large language model using methods such as model fine-tuning, RAG (Rapid Aspect Ratio), and prompt word engineering, applied to a webpage application. Based on this, the output answer can be understood as the response content returned by the intelligent agent to the target question.

[0060] The controller sends the target question to the browser. For example, the target question can be injected into the browser as a script using an automation tool to simulate user input; it can also send the question command generated based on the target question directly to the browser through a remote debugging protocol to control the behavior of the target webpage; and so on. This can be configured according to actual needs and is not specifically limited here.

[0061] In this way, after receiving the target question, the browser can send the target question to the agent through the target webpage, receive the agent's output answer for the target question through the target webpage, and then return the output answer to the controller so that the controller can perform subsequent operations based on the received output answer.

[0062] In this step, by simulating real user interaction, the submission of target questions and the acquisition of output answers can be achieved without the agent providing an open API. This is suitable for evaluating agents that do not provide an API.

[0063] In addition, optionally, since the automation tool in the example above cannot capture the output answer of the agent, the target script can be injected into the context of the target webpage so that the browser can capture the output answer located on the target webpage by executing the target script.

[0064] In this embodiment of the invention, the target script can be understood as a piece of JavaScript code, which can be used to listen to and extract the content output by the intelligent agent on the target webpage. Correspondingly, the context of the target webpage can be understood as the JavaScript execution environment currently loading the target webpage in the browser.

[0065] Injecting the target script into the context can be done in various ways, such as through remote debugging protocols or by automatically loading and executing it via a browser extension. These methods can be configured according to specific needs and are not limited here. In this way, the browser can capture the output answers located on the target webpage by executing the target script.

[0066] The above-mentioned alternative approach directly accesses the Document Object Model (DOM) of the target webpage through script injection, enabling real-time monitoring and capture of asynchronous streaming output content. This solves the problem that traditional crawlers cannot obtain complete output answers due to asynchronous updates of the target webpage, thus improving the accuracy and completeness of output answer capture.

[0067] Alternatively, there may be multiple target questions, which are sent to the browser sequentially. For the currently being processed target question among these multiple target questions, the target question is sent to the browser, including:

[0068] Get the random delay duration, and determine the target sending time of the current question based on the random delay duration and the previous sending time of the previous question. The previous question is the target question sent to the browser before the current question and adjacent to the current question.

[0069] The response is sent to the browser at the current moment, which is the moment the target is sent.

[0070] In the process of evaluating an agent based on multiple target questions, each target question is sent to the browser sequentially for processing. The random delay duration can be understood as a randomly generated time interval between two question transmissions, used to simulate user operation intervals. The previous transmission time can be understood as the specific time when the previous question was sent to the browser; the target transmission time can be understood as the specific time when the current question is planned to be sent to the browser, which can be determined by the sum of the previous transmission time and the random delay duration.

[0071] Furthermore, the current issue can be sent to the browser when the target sending time has arrived. Optionally, a timer can be used to poll whether the current time has reached the target sending time, or an event-driven mechanism can be used to trigger the target issue sending event at the target sending time; etc., these can be configured according to actual needs, and no specific limitations are made here.

[0072] The above-mentioned alternative schemes simulate the rhythm of real user input through a random delay mechanism, thereby effectively avoiding the detection of automated behavior by the agent based on behavior detection strategies. This ensures that the agent's evaluation process will not be interrupted or identified as abnormal access, thus improving the stability and concealment of the evaluation.

[0073] S130. Receive the output answer to evaluate the agent's question-answering performance by comparing the output answer with the target answer.

[0074] The controller compares the received output answer with the target answer, for example, by at least one of string matching, semantic similarity calculation, rule engine and scoring model, and evaluates the agent's question-answering performance based on the comparison results. In this embodiment of the invention, the question-answering performance can optionally characterize at least one of the accuracy, completeness and reliability of the agent's answer to the target question, which is related to the actual situation and is not specifically limited here.

[0075] In this step, the accuracy and error rates can be quickly calculated through automated comparison, quantifying the question-answering performance of the agent. Furthermore, it supports batch evaluation and continuous monitoring, thereby improving evaluation efficiency.

[0076] The technical solution of this invention involves a controller acquiring a target question for agent evaluation and a target answer configured for that question. The controller then sends the target question to a browser. The browser then sends the target question to the agent via the target webpage representing the agent, and receives the agent's output answer. The output answer is returned to the controller, which then compares the received output answer with the target answer to evaluate the agent's question-and-answer performance. This automated browser control facilitates the simulation of real user-agent interaction using the target webpage, enabling automated evaluation of the agent's question-and-answer performance. This is particularly suitable for automated evaluation of agents with non-publicly accessible APIs, customized through model fine-tuning, RAG (Rapid Application Generation), and prompt word engineering.

[0077] Based on this, an optional technical solution is that, when the browser's remote debugging mode is enabled and the target webpage is logged in, the controller establishes a connection with the browser through the remote debugging mode and reuses the session state of the target webpage after the connection is established.

[0078] Send the target question to the browser, including:

[0079] By leveraging the login state maintained by the session state, the target question is sent to the browser to evaluate the agent represented by the target webpage opened on the browser.

[0080] Remote debugging mode can be understood as a mode provided by the browser that allows external tools to control the browser's behavior through a remote debugging protocol. Session state can be understood as the session information of a user who has logged into the target webpage in the browser, which may include small text (cookies) and local storage, etc. This depends on the actual situation and is not specifically limited here.

[0081] Users can manually launch a browser and open the remote debugging port in remote debugging mode. After logging into the target webpage, the page remains open. At this time, the controller can establish a connection with the browser through remote debugging mode (specifically, the remote debugging port) and reuse the session state of the target webpage after the connection is established. This can avoid repeated logins for each evaluation, maintain the user's identity status, and ensure that the evaluation process is consistent with the real use scenario.

[0082] Login status can be understood as the user having completed identity authentication on the target webpage and remaining in an operational state. In this technical solution, the login status maintained by the session state can be used to send the target question to the browser to evaluate the intelligent agent represented by the target webpage opened on the browser. This ensures that the intelligent agent is in a normal service state during the evaluation process, avoiding access restrictions or abnormal behavior caused by not being logged in.

[0083] The above technical solution, by reusing session state, can avoid repeated logins, improve evaluation efficiency, and at the same time ensure that the evaluation environment is consistent with the real user environment, thereby enhancing the authenticity and reliability of the evaluation results.

[0084] Figure 3 This is a flowchart of another agent evaluation method provided in this embodiment of the invention. This embodiment is applicable to the automatic evaluation of agents, especially to the automatic evaluation of agents constructed based on model fine-tuning, RAG, and prompt word engineering. This method can be executed by the agent evaluation device provided in this embodiment of the invention. This device can be implemented in software and / or hardware, and can be configured on a browser, which can be integrated into a user terminal.

[0085] See Figure 3 The method of this invention specifically includes the following steps:

[0086] S210. Receive the target question sent by the controller for agent evaluation.

[0087] The controller can be understood as a program deployed on the control terminal, which is responsible for scheduling and evaluating tasks.

[0088] In this step, real-time communication between the controller and the browser is implemented to ensure that the target question is accurately transmitted to the target webpage.

[0089] S220. Send the target question to the agent via the target webpage, and receive the output answer from the agent in response to the target question via the target webpage.

[0090] For example, in this step, the actual user operation process can be completely simulated to ensure that the agent receives the target question and generates the corresponding output answer in normal interaction mode.

[0091] S230. Return the output answer to the controller so that the controller can evaluate the agent's question-answering performance by comparing the output answer with the target answer configured for the target question.

[0092] For example, the browser can extract the output answer via script and return it to the controller through a communication interface. It can also return the entire Hypertext Markup Language (HTML) structure of the answer area for the controller to parse and extract text, and so on. This can be configured according to actual needs and is not specifically limited here. This step enables automated return of the output answer, supports batch comparison and performance statistics by the controller, thereby improving evaluation efficiency.

[0093] The technical solution of this invention involves a browser receiving a target question from a controller for agent evaluation. Then, for the opened target webpage representing the agent, the browser sends the target question to the agent. The browser then receives the agent's output answer to the target question via the target webpage and returns it to the controller. The controller then evaluates the agent's question-answering performance by comparing the output answer with the target answer configured for the target question. This technical solution simulates the interaction between a real user and the agent via the target webpage, returning the agent's output answer to the controller. This allows for automated evaluation of the agent's question-answering performance in conjunction with the controller. It is particularly suitable for automated evaluation of agents with non-publicly accessible APIs, customized through model fine-tuning, RAG (Rapid Application Architecture), and prompt word engineering.

[0094] Based on this, an optional technical solution involves setting an input area and a send control on the target webpage, and sending the target question to the intelligent agent through the target webpage, including:

[0095] Trigger a location event for the input area to display the location result in the input area;

[0096] Trigger sequential input events for each second character in the target question to display each second character sequentially in the input area;

[0097] Trigger the activation event for the send control to send the target question, based on each second character, displayed in the input area to the agent.

[0098] In this context, the input area can be understood as the region on the target webpage where the user can input the target question. Correspondingly, the positioning event can be understood as an event that locates the input area, used to simulate the user's click or focus action on the input area. The browser triggers the positioning event to display the positioning result in the input area. For example, the browser can locate the input area element using a DOM selector and trigger the focus event, or it can simulate a mouse click event to give the input area focus, etc., without specific limitations here. This ensures that the input area is in an input-enabled state, simulating the starting point of a real user operation.

[0099] The second character can be understood as a character in the target question. Correspondingly, the sequential input event can be understood as the event that inputs each second character sequentially into the input area according to their order of appearance in the target question, simulating the user's behavior of typing character by character. The browser triggers the sequential input event to display each second character sequentially in the input area. For example, the browser can simulate character-by-character input through keyboard events, or it can achieve fast filling by setting the input area value property and triggering the input event, etc. This can be configured according to actual needs and is not specifically limited here.

[0100] The submit control can be understood as a control on the target webpage that can be used to submit a target question. Correspondingly, the activation event can be understood as an event that can start or enable the default function of the submit control to send the target question. In this technical solution, optionally, this event can be a click event, a swipe event, or a press event, etc., which can be set according to actual needs and is not specifically limited here. The browser can send the target question, based on each second character, displayed in the input area to the intelligent agent by triggering the activation event.

[0101] The above technical solution, by simulating the entire process of a real user on the target webpage from input to submission of the target question, can ensure the consistency between the intelligent agent evaluation process and human operation, and avoid the impact of abnormal operation on the validity and accuracy of the evaluation results.

[0102] Figure 4This is a flowchart of another intelligent agent evaluation method provided by an embodiment of the present invention. This embodiment is based on and optimized from the above-described technical solutions. In this embodiment, optionally, when the intelligent agent outputs each first character of the output answer for the target question sequentially to the target webpage, receiving the output answer of the intelligent agent for the target question through the target webpage includes: in response to each first character being output to the target webpage, obtaining the output answer on the target webpage based on each first character. The explanations of terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0103] See Figure 4 The method in this embodiment may specifically include the following steps:

[0104] S310. Receive the target question sent by the controller for agent evaluation.

[0105] S320. Send the target question to the agent via the target webpage.

[0106] S330. In response to the fact that the first character of each character in the output answer of the agent for the target question has been output to the target webpage in sequence, obtain the output answer on the target webpage based on each first character.

[0107] Here, the first character can be understood as a character in the output answer. This character can be at least one of Chinese characters, numbers, and symbols, depending on the specific situation, and no specific limitation is made here. Based on this, the output answer can be understood as the text composed of all the first characters.

[0108] In this embodiment of the invention, considering that the agent's answers are usually generated asynchronously and streamed by JavaScript, and thus asynchronously and streamed on the target webpage, the agent outputs and displays each first character in the order it appears in the output answer on the target webpage. Therefore, it is possible to monitor whether each first character constituting the output answer has been output to the target webpage in sequence, so as to obtain the complete output answer when that time is reached. Optionally, this can be achieved by listening to DOM change events in the answer area of ​​the target webpage to extract the complete output answer after the content stabilizes; alternatively, it can be achieved by polling to check whether the content in the answer area is no longer being updated, and then extracting the complete output answer when that time is reached; etc., without specific limitations.

[0109] S340. Return the output answer to the controller so that the controller can evaluate the agent's question-answering performance by comparing the output answer with the target answer configured for the target question.

[0110] The technical solution of this invention, by waiting for the intelligent agent to complete its output before obtaining the output answer, ensures that the obtained output answer is complete, avoiding the situation where the output answer is truncated or missing due to streaming output, thereby improving the consistency and accuracy of answer comparison.

[0111] Based on this, an optional technical solution, the above-mentioned agent evaluation method, further includes:

[0112] The first character output by the agent to the target webpage is obtained based on a preset time interval;

[0113] The content of adjacent first characters is compared to determine whether each first character is output to the target webpage.

[0114] The preset time interval can be understood as a pre-set time interval for polling and obtaining the first character that the agent has currently output to the target webpage. The number of these first characters can be one or more, depending on the actual output of the agent, and is not specifically limited here. By obtaining the first character based on the preset time interval, the output progress of the agent in outputting the answer can be monitored in real time, thereby supporting dynamic tracking of the streaming output process.

[0115] Furthermore, the content of adjacent first characters is compared, that is, the content of the first character obtained in the current round is compared with the first character obtained in the previous round. For example, text hashing or difference comparison algorithms can be used to detect content changes. Then, based on this, it can be determined whether each first character in the output answer has been output to the target webpage, that is, whether the agent has output a complete output answer. By judging whether the output is complete through content changes, it is possible to avoid relying on fixed waiting time and improve capture efficiency and accuracy.

[0116] The above technical solution, through timed polling and content comparison mechanisms, enables intelligent monitoring of the agent's streaming output process, ensuring that the complete answer is captured in a timely manner after output is completed, and reducing invalid waiting time.

[0117] Optionally, the adjacent first characters obtained may include the current first character and the previous first character. A content comparison is performed on the adjacent first characters to determine whether each first character is output to the target webpage, including:

[0118] Compare the content of the currently obtained first character with the previous first character;

[0119] If the current first character differs from the previous first character, the timer is restarted; otherwise, the timer continues to run. The timing of each first character is determined by whether the timer duration meets the preset duration condition.

[0120] Here, the current first character can be understood as the first character obtained in the most recent polling, and the previous first character can be understood as the first character obtained in the previous polling. In this case, the content comparison of adjacent first characters described above is the content comparison of the current first character and the previous first character.

[0121] Furthermore, the timer can be understood as a timing tool used to record the duration of a period when the content remains unchanged. In this technical solution, optionally, the timer can also be called a stable timer. The preset duration condition can be understood as a pre-set condition related to duration that represents the completion of the agent's output. In this technical solution, optionally, it can be represented by the timer's timing duration being ≥ the preset duration threshold.

[0122] If the current first character is different from the previous first character, it means that the agent is still outputting, and the timer can be restarted. Otherwise, it means that the agent may have finished outputting, and the timer can be controlled to continue outputting, i.e., the timer counts up. Thus, the output answer can be judged to be completely output to the target webpage by whether the timer's duration meets the preset duration condition.

[0123] The above technical solution, combined with content change detection and timer mechanisms, can avoid misjudging the agent's output as complete due to short pauses in the agent's output. This achieves accurate determination of content stability, thereby improving the accuracy and reliability of capturing the output answer.

[0124] Based on this, in order to better understand the technical solutions in the embodiments of the present invention as a whole, the following example of agent response (i.e. output answer) integrity monitoring will be used for illustrative purposes.

[0125] For example, see Figure 5 The specific implementation process is as follows: Extract the latest output text (i.e., the current first character) and compare its content with the output text of the previous cycle (i.e., the previous first character); if the content has changed, reset the timer to start counting again; otherwise, check if the timer has already started, and if so, continue counting; otherwise, start the timer to begin counting; further, if the counting duration is greater than or equal to a preset duration threshold, a complete output answer can be returned; otherwise, the above steps can be repeated after a preset time interval until a complete output answer can be returned. The above example demonstrates the complete capture of the output answer in streaming output.

[0126] Figure 6 This is a structural block diagram of an agent evaluation system provided in an embodiment of the present invention. This embodiment is applicable to the automatic evaluation of agents, and is particularly applicable to the automatic evaluation of agents constructed based on model fine-tuning, RAG, and cue word engineering.

[0127] See Figure 6 The intelligent agent evaluation system described in this embodiment of the invention may include: a controller 410 and a browser 420 communicatively connected to the controller 410, wherein a target webpage representing a web-based intelligent agent is opened on the browser 420; wherein,

[0128] Controller 410 is used to obtain the target question for evaluating the agent and the target answer configured for the target question, and send the target question to browser 420;

[0129] Browser 420 is used to receive the target question, send the target question to the intelligent agent through the target webpage, receive the output answer of the intelligent agent in response to the target question through the target webpage, and return the output answer to the controller 410;

[0130] The controller 410 is also used to receive output answers in order to evaluate the agent’s question-answering performance by comparing the output answers with the target answers.

[0131] The technical solution of this invention, through the division of labor and cooperation between the controller and the browser, can simulate the interaction process between a real user and an intelligent agent by using the target webpage, thereby realizing the automated evaluation of the intelligent agent's question-answering performance. It is especially suitable for the automated evaluation of intelligent agents whose APIs are not publicly available and have been customized through model fine-tuning, RAG and prompt word engineering.

[0132] Building upon this, to better understand the various technical solutions described above, the following example illustrates the concepts. For instance, web-based intelligent agents driven by large language models (such as enterprise intelligent customer service and AI question-answering platforms) are widely deployed in browser environments, and these agents no longer expose their APIs. Therefore, in this example, the controller evaluates its question-answering performance by interacting with the front-end user page (i.e., the target webpage described above). The specific implementation process is as follows:

[0133] 1. Evaluation Dataset Preparation

[0134] In this example, question-answering performance can be evaluated by setting subjective and / or objective questions. Here, we take setting objective questions as an example. Each data entry contains a question identifier (id), a target question (question), options (option), and a target answer (answer) configured for the question.

[0135] 2. Environmental preparation and connection

[0136] 2.1 Launch your browser and enable remote debugging mode;

[0137] 2.2 Users manually log in to the target webpage;

[0138] 2.3 The controller establishes a connection with a browser that has enabled remote debugging mode through automation tools, and reuses the session state of the target webpage after the connection is established to maintain the user's login status on the target webpage.

[0139] 3. Target problem loading and preprocessing

[0140] 3.1 Read the question and option from the predefined evaluation dataset;

[0141] 3.2 Concatenate the question and option, and add a custom prompt to form an instruction text: "Please briefly answer the following questions: {question} Options: {A. ... B. ...}", and send it to the browser.

[0142] 4. Target issue injection and sending

[0143] 4.1 Locate the input area in the target webpage, that is, trigger the location event for the input area, which is equivalent to simulating a user clicking on the input area;

[0144] 4.2 Trigger the sequential input event for each second character in the target question, which is equivalent to simulating a user inputting each second character sequentially by typing on the keyboard;

[0145] 4.3 Trigger the activation event for the send control, which is equivalent to simulating the user clicking the send control to submit the target question and start the agent's inference.

[0146] 5. Monitoring the integrity of agent responses

[0147] 5.1 Initialization: Record the latest output text (i.e., the current first character as described above);

[0148] 5.2 Dynamic Polling: Perform the following operations at 0.5-second intervals:

[0149] a. Accurately extract the latest output text of the agent (i.e., the current first character as described above) through the target script (i.e., JavaScript script), and then perform subsequent steps b and c;

[0150] b. Compare the latest output text with the output text of the previous cycle (i.e., the previous first character as described above): If the content is different (i.e. the content has changed), the stabilization timer can be reset and the latest output text can be recorded; otherwise, the stabilization timer continues to count and it is determined whether it is ≥4 seconds.

[0151] c. If the content remains stable for ≥4 seconds and the length of the latest recorded output text is ≥5 characters, then the agent is deemed to have completed its response;

[0152] 5.3 Timeout Protection: The total waiting time is ≤30 seconds. If the timeout occurs, the latest recorded output text will be returned.

[0153] Considering that agent responses are usually generated asynchronously and streamed by JavaScript, and thus displayed asynchronously on the target webpage, traditional static crawlers cannot reliably capture the complete output of the agent, i.e., the final output answer. Therefore, this example presents a dynamic response capture method based on content stability, as shown in step 5. It determines whether the agent response is complete by monitoring changes in the content of a specific DOM, thereby solving the problem of obtaining dynamically loaded content.

[0154] Furthermore, considering that the agent's output answer depends on the context history, this example fully simulates the real interaction process of user inputting the target question, waiting for the agent's response, and reading the output answer.

[0155] 6. Output answer persistence and flow control

[0156] 6.1 Save the target question, options, and output answer to a JSON file. The new JSON file can contain five fields: id, question, option, answer, and output answer.

[0157] 6.2 Write each piece of evaluation data to disk as soon as it is completed;

[0158] 6.3 Random delays are inserted between target problems to prevent agents from detecting automated behavior through behavior detection strategies (such as detecting input speed and mouse movement trajectory).

[0159] In this example, a large number of predefined question sets can be submitted to the agent without human intervention, and the agent's response to each target question in the question set can be automatically recorded.

[0160] 7. Evaluation Results Generation

[0161] 7.1 Count the number of successful / failed evaluation data entries;

[0162] 7.2 Extract the options from the output field of the output JSON file;

[0163] 7.3 Compare the options extracted for each target question in 7.2 with the corresponding answer field, and output the ratio of the matches to all as the evaluation result.

[0164] The above example demonstrates that, without modifying the source code, the browser can be automatically controlled to simulate real user operations, thereby automating the question-answering performance evaluation of web-based intelligent agents. This method can meet the evaluation needs of intelligent agents built using methods such as model fine-tuning, RAG, and prompt word engineering. Moreover, it is a low-intrusion, highly robust, and highly general evaluation method.

[0165] Figure 7 This is a structural block diagram of an agent evaluation device provided in an embodiment of the present invention. This device is used to execute the agent evaluation method provided in any of the above embodiments. This device and the agent evaluation methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the agent evaluation device can be found in the embodiments of the above agent evaluation methods. See also... Figure 7 The device is configured on a controller, which is connected to a browser. The browser opens a target webpage that represents a web-based intelligent agent. The device may include: a target answer acquisition module 510, a target question sending module 520, and an intelligent agent evaluation module 530.

[0166] The target answer acquisition module 510 is used to acquire the target question for agent evaluation and the target answer configured for the target question.

[0167] The target question sending module 520 is used to send the target question to the browser, so that the browser can send the target question to the intelligent agent through the target webpage, and receive the output answer of the intelligent agent in response to the target question through the target webpage, and return the output answer;

[0168] The agent evaluation module 530 is used to receive the output answer in order to evaluate the agent's question-answering performance by comparing the output answer with the target answer.

[0169] Optionally, if the browser's remote debugging mode is enabled and the user is logged into the target webpage, the controller establishes a connection with the browser through remote debugging mode and reuses the target webpage's session state after the connection is established. The target issue sending module 520 may include:

[0170] The first sending unit is used to send the target question to the browser by means of the login state maintained by the session state, so as to evaluate the agent represented by the target webpage opened on the browser.

[0171] Optionally, there may be multiple target questions, which are sent to the browser sequentially. The target question sending module 520, which handles the current target question among the multiple target questions, may include:

[0172] The target sending time determination unit is used to obtain the random delay duration and determine the target sending time of the current question based on the random delay duration and the previous sending time of the previous question. The previous question is the target question sent to the browser before and adjacent to the current question.

[0173] The second sending unit is used to send the current question to the browser in response to the fact that the current time is the target sending time.

[0174] Optionally, the aforementioned agent evaluation device may further include:

[0175] The target script injection module is used to inject target scripts into the context of a target webpage, so that the browser can capture the output answers located on the target webpage by executing the target scripts.

[0176] The agent evaluation device provided in this invention involves a controller that acquires a target question and a configured target answer for agent evaluation via a target answer acquisition module. Then, a target question sending module sends the target question to a browser. The browser then sends the target question to the agent via a target webpage representing the agent, and receives and returns the agent's output answer. Finally, the agent evaluation module compares the received output answer with the target answer to evaluate the agent's question-and-answer performance. This device, through automated browser control, facilitates the simulation of real user-agent interaction via a target webpage, thereby enabling automated evaluation of the agent's question-and-answer performance. It is particularly suitable for automated evaluation of agents with non-publicly accessible APIs, customized through model fine-tuning, RAG (Rapid Application Group) engineering, and prompt word engineering.

[0177] The agent evaluation device provided in the embodiments of the present invention can execute the agent evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0178] It is worth noting that in the embodiments of the above-mentioned intelligent agent evaluation device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0179] Figure 8This is a structural block diagram of another agent evaluation device provided in an embodiment of the present invention. This device can be used to execute the agent evaluation method provided in any of the above embodiments. This device and the agent evaluation methods of the above embodiments belong to the same inventive concept. Details not described in detail in the embodiments of the agent evaluation device can be referred to the embodiments of the above agent evaluation methods. See also Figure 8 The device is configured in a browser, which communicates with the controller. A target webpage representing a web-based intelligent agent is opened on the browser. The device may include: a target question receiving module 610, an output answer receiving module 620, and an output answer return module 630.

[0180] The target question receiving module 610 is used to receive the target question sent by the controller for agent evaluation;

[0181] The output answer receiving module 620 is used to send the target question to the intelligent agent through the target webpage, and to receive the output answer output by the intelligent agent in response to the target question through the target webpage.

[0182] The output answer return module 630 is used to return the output answer to the controller so that the controller can evaluate the agent's question-answering performance by comparing the output answer with the target answer configured for the target question.

[0183] Optionally, when the agent outputs the first character of each character in the output answer to the target question sequentially onto the target webpage, the output answer receiving module 620 may include:

[0184] The output answer acquisition unit is used to acquire the output answer based on each first character on the target webpage in response to the fact that each first character has been output to the target webpage.

[0185] Optionally, the aforementioned intelligent agent evaluation device may further include:

[0186] The first character acquisition module is used to acquire the first character output by the agent to the target webpage based on a preset time interval before all first characters are output to the target webpage.

[0187] The first character output judgment module is used to compare the content of adjacent first characters to determine whether each first character is output to the target webpage.

[0188] Based on this, optionally, the adjacent first character obtained may include the current first character and the previous first character, and the first character output judgment module may include:

[0189] The content comparison unit can be used to compare the content of the currently obtained first character with the previous first character;

[0190] The first character output judgment unit is used to respond to the fact that the current first character is different from the previous first character. It controls the timer to restart the countdown, otherwise it controls the timer to continue the countdown. The timer's countdown duration is used to determine whether each first character has been output to the target webpage.

[0191] Optionally, an input area and a send control are set on the target webpage, and the answer receiving module 620 may include:

[0192] The positioning event triggering unit is used to trigger positioning events for the input area so as to display the positioning results in the input area;

[0193] The sequential input event triggering unit is used to trigger sequential input events for each second character in the target question, so as to display each second character sequentially in the input area;

[0194] An activation event triggering unit is used to trigger an activation event for the send control, so as to send the target question, which is displayed in the input area and consists of each second character, to the agent.

[0195] The agent evaluation device provided in this invention allows a browser to receive a target question for agent evaluation sent by a controller via a target question receiving module. Then, via an output answer receiving module, the browser sends the target question to the agent through the target webpage representing the agent on the webpage. The browser then receives the agent's output answer in response to the target question via the target webpage and returns the output answer to the controller via an output answer return module. The controller then evaluates the agent's question-and-answer performance by comparing the output answer with the target answer configured for the target question. This device simulates the interaction between a real user and the agent via a target webpage, and returns the agent's output answer to the controller in conjunction with the controller to automate the agent's question-and-answer performance evaluation. It is particularly suitable for the automated evaluation of agents whose APIs are not publicly available and have been customized through model fine-tuning, RAG (Rapid Application Group) engineering, and prompt word engineering.

[0196] The agent evaluation device provided in the embodiments of the present invention can execute the agent evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the method execution.

[0197] It is worth noting that in the embodiments of the above-mentioned intelligent agent evaluation device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0198] Figure 9 A schematic diagram of a control terminal or user terminal (hereinafter collectively referred to as terminal device) 10, which can be used to implement embodiments of the present invention, is shown. Terminal device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Terminal device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0199] like Figure 9 As shown, the terminal device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the terminal device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0200] Multiple components in terminal device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows terminal device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0201] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as agent evaluation methods.

[0202] In some embodiments, the agent evaluation method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on terminal device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the agent evaluation method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the agent evaluation method by any other suitable means (e.g., by means of firmware).

[0203] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips or system-on-a-chips (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0204] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0205] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0206] To provide interaction with a user, the systems and techniques described herein can be implemented on a terminal device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the terminal device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0207] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0208] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0209] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication unit 19, or installed from storage unit 18, or installed from ROM 12. When the computer program is executed by processor 11, it performs the functions defined in the methods of the embodiments of the present invention.

[0210] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0211] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for evaluating an intelligent agent, characterized in that, The method is applied in a controller that is communicatively connected to a browser, where a target webpage representing a web-based intelligent agent is open. Obtain the target question for evaluating the agent and the target answer configured for the target question; The target question is sent to the browser, so that the browser sends the target question to the agent through the target webpage, receives the output answer from the agent for the target question through the target webpage, and returns the output answer; The output answer is received to evaluate the agent's question-answering performance by comparing the output answer with the target answer.

2. The method according to claim 1, characterized in that, When the remote debugging mode of the browser is enabled and the target webpage is logged in, the controller establishes a connection with the browser through the remote debugging mode and reuses the session state of the target webpage after the connection is established. Sending the target question to the browser includes: Using the login state maintained by the session state, the target question is sent to the browser to evaluate the agent represented by the target webpage opened on the browser.

3. The method according to claim 1, characterized in that, The number of target questions is multiple, and these multiple target questions are sent to the browser sequentially. Sending the target question to the browser, for the currently processed target question among the multiple target questions, includes: Obtain a random delay duration, and determine the target sending time of the current question based on the random delay duration and the previous sending time of the previous question, wherein the previous question is the target question sent to the browser before and adjacent to the current question; In response to the current time being the time the target was sent, the current question is sent to the browser.

4. The method according to claim 1, characterized in that, Also includes: The target script is injected into the context of the target webpage so that the browser can capture the output answer located on the target webpage by executing the target script.

5. A method for evaluating an intelligent agent, characterized in that, The method is applied in a browser, which is communicatively connected to a controller, and a target webpage representing a web-based intelligent agent is opened on the browser. The method includes: Receive the target question sent by the controller for evaluating the agent; The target question is sent to the intelligent agent through the target webpage, and the output answer of the intelligent agent for the target question is received through the target webpage. The output answer is returned to the controller, so that the controller evaluates the agent's question-answering performance by comparing the output answer with the target answer configured for the target question.

6. The method according to claim 5, characterized in that, When the agent sequentially outputs the first character of its output answer to the target question onto the target webpage, receiving the output answer from the agent for the target question via the target webpage includes: In response to each of the first characters being output to the target webpage, the output answer based on each of the first characters on the target webpage is obtained.

7. The method according to claim 6, characterized in that, Before the response is completed and each of the first characters is output to the target webpage, the method further includes: The first character output by the intelligent agent to the target webpage is obtained based on a preset time interval; The content of adjacent first characters is compared to determine whether each first character is output to the target webpage.

8. The method according to claim 7, characterized in that, The adjacent first characters obtained include the current first character and the previous first character. The step of comparing the content of the adjacent first characters to determine whether each first character is output to the target webpage includes: The content of the currently obtained first character is compared with that of the previous first character; In response to the fact that the current first character is different from the previous first character, the timer is controlled to restart; otherwise, the timer is controlled to continue timing, so as to determine whether each first character has been output to the target webpage by whether the timing duration of the timer meets the preset duration condition.

9. The method according to claim 5, characterized in that, The target webpage has an input area and a send control. Sending the target question to the intelligent agent through the target webpage includes: Trigger a location event for the input area to display the location result in the input area; Trigger sequential input events for each second character in the target question to sequentially display each second character in the input area; Trigger an activation event for the send control to send the target question, displayed in the input area and composed of each of the second characters, to the agent.

10. An intelligent agent evaluation system, characterized in that, include: A controller and a browser communicatively connected to the controller, wherein a target webpage representing a web-based intelligent agent is opened on the browser; wherein... The controller is configured to acquire a target question for evaluating the agent and a target answer configured for the target question, and send the target question to the browser; The browser is configured to receive the target question, send the target question to the intelligent agent through the target webpage, receive the output answer output by the intelligent agent for the target question through the target webpage, and return the output answer to the controller; The controller is also configured to receive the output answer in order to evaluate the agent's question-answering performance by comparing the output answer with the target answer.

Images