Method and apparatus for generating evaluation data for a graphical interface agent

Abstract

This specification provides a method and apparatus for generating evaluation data for a graphical user interface (GUI) agent. The method involves comparing the execution trajectory obtained by the GUI agent executing user commands with a pre-constructed optimal reference trajectory to extract corresponding evaluation features. These features include indicator values ​​for various predetermined evaluation metrics. By mapping each indicator value in the evaluation features to the same value domain, respective evaluation scores for each predetermined evaluation metric can be obtained. Based on these scores, traceable evaluation data can be generated. This allows for the evaluation of the GUI agent from multiple dimensions, facilitating comparisons between multiple GUI agents or targeted optimization of the GUI agent. By setting comprehensive evaluation metrics and independent mapping of indicator values ​​to the same value domain, a more accurate and stable evaluation of the GUI agent can be achieved through a precise and traceable evaluation system.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to a method and apparatus for generating evaluation data for graphical interface intelligent agents. Background Technology

[0002] An intelligent agent is generally defined as an entity capable of perceiving its environment and taking actions to achieve specific goals. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactivity. By perceiving changes in the environment, intelligent agents utilize learned knowledge and algorithms to make judgments and decisions, thereby executing actions to influence the environment or achieve predetermined goals. For example, a machine learning model can act as an intelligent agent to process input data and complete related tasks, or to classify corresponding entities; a robot, as an intelligent agent, can perform related actions according to human instructions, and so on.

[0003] With the rapid development of artificial intelligence technology, intelligent agents are increasingly involved in a wide range of fields. Graphical User Interface (GUI) agents typically refer to intelligent systems that simulate the interaction between human users and a graphical user interface (GUI) to complete complex tasks such as web page operations, document processing, application management, cross-device collaboration, and pre-defined actions (e.g., hailing a ride). These agents can understand user instructions and execute operations such as clicking, inputting, and swiping the screen to automate tasks, reduce user operations, and improve user experience. GUI agents have become an important technological direction in fields such as human-computer interaction, automated office work, and intelligent operation and maintenance. Unlike traditional single-step API call-based agents, GUI agents have characteristics such as multi-step dependencies, tool call correlations, and logically coherent operations. The quality of task completion depends not only on the final result but also on the rationality of the execution trajectory. Therefore, how to evaluate GUI agents is a crucial technical issue in their application. Summary of the Invention

[0004] This specification describes one or more embodiments of a method and apparatus for generating evaluation data for a graphical user interface agent, in order to solve one or more problems mentioned in the background art.

[0005] According to a first aspect, a method for generating evaluation data for a graphical user interface (GUI) agent is provided. The method includes: acquiring a first execution trajectory obtained by the GUI agent executing a first user instruction, the first user instruction corresponding to a first task type; extracting first evaluation features from the first execution trajectory based on a comparison with a pre-constructed first reference trajectory for multiple predetermined evaluation indicators, the first evaluation features including indicator values ​​on each predetermined evaluation indicator, the first reference trajectory being obtained based on the first task type; mapping each indicator value in the first evaluation features to the same value domain to obtain each evaluation score corresponding to each predetermined evaluation indicator; and generating first evaluation data based on each evaluation score.

[0006] In some embodiments, extracting a first evaluation feature from the first execution trajectory for multiple predetermined evaluation metrics includes at least one of the following: determining the number of operands in the first execution trajectory that deviate from the optimal operation path corresponding to the first reference trajectory; determining a metric value for the logical jump number of the predetermined evaluation metrics based on the operands; determining the number of correct steps for operation elements in the first execution trajectory that are correctly identified and have correct operation coordinates; determining a metric value for the element identification accuracy of the predetermined evaluation metrics based on the ratio of the number of correct steps to the total number of steps in the first execution trajectory; determining the number of correction steps for performing correction operations in the first execution trajectory; and determining a metric value for the anomaly handling rate of the predetermined evaluation metrics based on the ratio of the number of correction steps to the number of abnormal operands in the first execution trajectory.

[0007] In some further embodiments, the plurality of predetermined evaluation metrics further include at least one of the following: error operation rate, which is the ratio of the number of erroneous operations in the first execution trajectory to the total number of operations; number of missing critical steps, which is the number of critical steps missing in the first execution trajectory relative to the critical step set determined based on the first reference trajectory; number of erroneous tool calls, which is the number of steps in the first execution trajectory that did not call the corresponding effective tools; and redundancy rate, which is the number of steps N in the first execution trajectory. total Optimal number of steps corresponding to the first reference trajectory The difference between The ratio of execution time to execution time; the time consumption ratio is the ratio of the execution time of the first execution trajectory to the optimal execution time T corresponding to the first reference trajectory. opt The ratio; path matching degree, which is the ratio of the steps in the first execution trajectory to the optimal path in the first reference trajectory. The number of consistent steps, compared to the total number of steps in the first execution trajectory. The ratio of the first execution trajectory to the total number of steps; the environment adaptation rate is the ratio of the number of steps in the first execution trajectory where the operation element is a valid element determined based on the first reference trajectory and the operation position is correct; the correction step count is the number of correction operation steps in the first execution trajectory; the number of irrelevant resource occupancy is the number of unnecessary processes opened during the determination of the first execution trajectory.

[0008] In some embodiments, the first reference trajectory is constructed by: for a first candidate task under a first task type, determining the following labeled data: task description, task type, and target operation result; obtaining the operation sequence of manually executing the first candidate task as the target operation sequence; and generating the first reference trajectory based on the target operation sequence.

[0009] In some further embodiments, the target operation sequence includes each atomic operation and each target element corresponding to each atomic operation. Generating the first reference trajectory based on the target operation sequence further includes: statistically analyzing trajectory parameters based on the target operation sequence, including at least one of the following: the optimal number of steps describing the number of atomic operations, the optimal time determined based on the task completion time, the optimal operation path described by sub-goals achieved by merging one or more atomic operations, and effective operation elements including each target element; forming the first reference trajectory based on the trajectory parameters and the target operation sequence.

[0010] In some embodiments, before extracting a first evaluation feature from the first execution trajectory based on a comparison with a pre-constructed first reference trajectory, the method further includes performing at least one of the following preprocessing operations on the first execution trajectory: removing invalid operations and standardizing operation elements.

[0011] In some embodiments, each index value in the first evaluation feature is mapped to the same value range to obtain each evaluation score corresponding to each predetermined evaluation index, including: for a single predetermined evaluation index, based on predetermined rules, the index value is mapped to the corresponding score in the same value range according to the size condition it meets, and the predetermined rules describe the correspondence between the index value size and the score.

[0012] In some embodiments, generating first evaluation data based on each evaluation score includes: determining a fusion score of the first execution trajectory based on a weighted sum of the evaluation scores; and outputting the first evaluation data by a large language model based on the evaluation scores and the fusion score.

[0013] In some further embodiments, each predetermined evaluation index corresponds to multiple index categories, and each index category corresponds to a weighted weight for each category. The fusion score is determined based on the weighted sum of the various evaluation scores, including: determining the average evaluation score of the evaluation index corresponding to each index category; and weighting and summing the average evaluation scores according to the weighted weights of each category to obtain the fusion score.

[0014] In some embodiments, generating first evaluation data based on various evaluation scores includes: fusing the evaluation scores of various execution trajectories, including the first execution trajectory, on each predetermined evaluation index by means of summation, averaging, or statistical value to obtain various fused evaluation parameters; and generating evaluation data of the agent using the various fused evaluation parameters.

[0015] In some embodiments, mapping each index value in the first evaluation feature to the same value range to obtain each evaluation score corresponding to each predetermined evaluation index includes: detecting whether each evaluation index meets predetermined conditions, the predetermined conditions including: the average score of the internal evaluation index of the index category is lower than a predetermined value, or, there is an extreme value, or, an abnormal evaluation index using an alternative solution, the alternative solution is determined by extended equivalence mapping or semantic similarity; for evaluation indicators that meet the predetermined conditions, the corresponding evaluation score is determined by manual review.

[0016] According to a second aspect, an apparatus for generating evaluation data for a graphical user interface (GUI) agent is provided. The apparatus includes: an acquisition unit configured to acquire a first execution trajectory obtained by the GUI agent executing a first user instruction, the first user instruction corresponding to a first task type; an extraction unit configured to extract first evaluation features from the first execution trajectory based on a comparison with a pre-constructed first reference trajectory for multiple predetermined evaluation indicators, the first evaluation features including indicator values ​​on each predetermined evaluation indicator, the first reference trajectory being acquired based on the first task type; a mapping unit configured to map each indicator value in the first evaluation features to the same value domain to obtain each evaluation score corresponding to each predetermined evaluation indicator; and an evaluation unit configured to generate first evaluation data based on each evaluation score.

[0017] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0018] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0019] The methods and apparatus provided in the embodiments of this specification allow for the extraction of corresponding evaluation features by comparing the execution trajectory obtained from executing user instructions on a GUI agent with a pre-constructed optimal reference trajectory. These evaluation features include indicator values ​​for various predetermined evaluation metrics. By mapping each indicator value in the evaluation features to the same value domain, respective evaluation scores for each predetermined evaluation metric can be obtained. Based on these evaluation scores, traceable evaluation data can be generated. In this way, the GUI agent can be evaluated from multiple dimensions, which can be used for comparison of multiple GUI agents or for targeted optimization of the GUI agent. By setting comprehensive evaluation metrics and independent mapping of indicator values ​​to the same value domain, a more accurate and stable evaluation of the GUI agent can be achieved through a precise and traceable evaluation system. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This diagram illustrates the token processing flow and generation principle of text generated word by word using a large language model.

[0022] Figure 2 This document illustrates a specific application scenario.

[0023] Figure 3 This diagram illustrates a process for generating evaluation data for a graphical user interface agent according to one embodiment.

[0024] Figure 4 A schematic block diagram of an apparatus for generating evaluation data for a graphical user interface agent according to one embodiment is shown. Detailed Implementation

[0025] First, we will explain some of the terms or concepts that may be used in this article.

[0026] Large Language Models (LLMs), also known simply as large models, are natural language processing models based on deep learning techniques. Their parameter count typically ranges from billions to hundreds of billions or even higher, possessing powerful language understanding and generation capabilities. LLMs can employ the Transformer architecture or its variants (such as GPT and BERT), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing exceptionally well in natural language tasks. By pre-training on large-scale corpora, LLMs learn the statistical features and semantic relationships of language, enabling them to generalize effectively. The core capabilities of LLMs include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Their usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, users design prompts to guide the LLM in generating specific outputs. These prompts can be textual descriptions of the task or instructions used to stimulate the LLM's semantic understanding and generation capabilities. In fine-tuning mode, large language models are further trained on small-scale datasets within a specific domain to optimize their performance on specific tasks. The powerful generalization capabilities and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0027] Multimodal Large Language Models (MLLMs) typically refer to large language models capable of simultaneously processing and understanding multiple types of data (such as text, images, audio, and video). By integrating information from different modalities, MLLMs achieve more comprehensive perception, reasoning, and generation capabilities, and are one of the important research directions in the field of artificial intelligence.

[0028] The token processing and generation principles of large language models are as follows: Figure 1 As shown. Reference Figure 1A large language model can contain at least multiple decoding layers (and may include encoding layers before the decoding layers). For the input prompt, `Prompt`, after discretization encoding (tokenizer), it can be divided into `m` "prompt tokens". Each "prompt token" represents the discretized prompt information (such as a word) within the prompt word. For a multimodal large model (MLLM), the prompt information can be at least one of text, image, or audio. After embedding, each prompt token corresponds to a corresponding embedding vector, which is further processed by the large language model. Each decoding layer can contain encoded representations obtained by encoding (or decoding) the `m` prompt tokens, as well as hidden representations for predicting the next word (a single word in the vocabulary, etc.). The hidden representation of the next word is `h`. a t (a = 1, 2, ..., N). After encoding in the last decoding layer N, the prediction layer maps the hidden representation of the next word to a probability distribution p of each word (a word list character, which is a token used as a candidate element). Using the probability distribution p, the words with the highest probabilities can be selected as the predicted next word. For example, the single word with the highest probability can be selected as the predicted next word. The predicted next word can be added to the historical predicted word sequence, which, along with the original prompt information, can be used as input to the large language model to predict subsequent words. This process is iterated over multiple prediction cycles until the end-of-line character is predicted, and the final word sequence is the generated text.

[0029] User graphical interface (GUI) data includes both image and structural data. A GUI is typically a graphical interface used for interaction between a terminal device and the user, displayed to the user and allowing them to interact with elements such as graphics. Image data can be obtained through page screenshots or snapshots. Snapshots usually capture the entire screen, while page screenshots can be partial or multi-page (long) screenshots. Structural data includes data describing the page structure, such as the Document Object Model (DOM).

[0030] GUI agents, which can be implemented based on multimodal large models, can utilize user interface data to predict and execute corresponding operations on various user interfaces through various prediction schemes, resulting in execution sequences (or execution trajectories). Various prediction schemes include: end-to-end mapping, which treats the GUI agent as a policy network, taking the current screenshot of the user interface (visual information) and user commands (linguistic information) as input, and outputting the specific operation to be executed next (such as clicking, inputting text, etc.); a thought chain-based scheme, where first, a more powerful multimodal large model (MLLM) generates a natural language text describing the thought process, i.e., a thought chain (CoT), based on the screenshot and user commands; then, the CoT is used as context information and input to the GUI agent, allowing it to predict the operations to be executed on the user interface based on the CoT; and so on.

[0031] The Model Context Protocol (MCP) is an open protocol for connecting Large Language Model (LLM) applications with external data sources and tools. An LLM application can include one or more agents. The agent, acting as the brain of the LLM, understands natural language tasks and performs tasks such as task planning and tool invocation. An LLM application can be an application of the LLM workflow paradigm, which may include multiple pre-defined or agent-planned operational steps.

[0032] MCP provides a unified, machine-readable description of all external services (such as tools and databases), essentially providing a specification for the interaction between agents in LLM applications and external services. The MCP architecture includes the MCP host, MCP clients, and the MCP server. The MCP host typically refers to the AI ​​application and is the initiator of the interaction, such as a GUI agent. The MCP client resides within the MCP host and is used to discover services provided by the MCP server (such as services provided through tools) and to exchange information between the LLM and the MCP server. The MCP server provides external services to the LLM. Specifically, the MCP server connects to external systems such as databases and web services, receives service call requests from the MCP client, forwards these requests to the external systems, receives responses from the external systems, and forwards the responses back to the MCP client, thus providing diverse functionalities for the LLM.

[0033] The technical solutions provided in this specification are described below with reference to the accompanying drawings.

[0034] Figure 2 A schematic diagram of a specific implementation scenario of this specification is shown. Figure 2The implementation scenario is a map application. In a map-based user interface, the user enters the command "Help me hail a taxi to Zhongguancun" in the interaction box. At this time, the map application can obtain user interface information, such as screenshots or page information before the current page is rendered (including page layout and content, such as the DOM tree). Figure 2 The screenshot shows that to complete the task of "hailing a taxi to Zhongguancun," the map application needs to perform a series of operations, such as entering the destination "Zhongguancun," clicking the "hail a taxi" button, and so on. For example, the next operation after entering "Help me hail a taxi to Zhongguancun" is to click the "hail a taxi" button (e.g., ...). Figure 2 The button or location marked by box 201 in the image allows users to hail a ride based on their current location and destination. Those skilled in the art will understand that after clicking the "hail a ride" button, the application may redirect to a new screen requiring further action. In this case, the subsequent action needs to be performed in conjunction with the new page and the target task in the user's instructions. For example... Figure 3 In the app, after clicking the ride-hailing button, you will be redirected to the route display and vehicle selection interface. Clicking the "Agree to Agreement" button will complete the vehicle hailing process.

[0035] In map applications, the process of predicting and executing operations (such as clicking a ride-hailing button or entering text like a phone number) based on screenshots and user commands can be accomplished by a GUI agent. This GUI agent can be a multimodal large model or other intelligent agents (such as a processor embedded with a multimodal large model), and is not limited here. The GUI agent can be deployed on the client-side or server-side of the map application, and this specification does not limit its deployment thereto. An example of the operation path predicted by the GUI agent is: "Enter the command 'Help me hail a ride to Zhongguancun' in the search box — Click the 'Ride-hailing button' — Click the 'Agree to Agreement' button…".

[0036] To accurately assess the predictive performance of a GUI agent, its prediction results (such as operation paths) can be evaluated. Conventional techniques typically employ a "results-oriented" paradigm for evaluating GUI agents, judging performance solely by whether the task was completed (e.g., whether a ride-hailing operation was successfully completed) or whether the result was correct (e.g., whether the ride reached the correct destination). This evaluation method may have the following problems: the process is uninterpretable, making it impossible to pinpoint issues such as "correct result but redundant process" or "incorrect result but only missing a single key step," hindering guidance for iterative optimization of the GUI agent; the evaluation dimension is singular, failing to distinguish between lucky task completion and robust task completion; evaluation criteria are individualized: lacking structured evaluation criteria, different evaluators exhibit significant biases in judging "reasonable trajectories," making it unfriendly for cross-agent horizontal comparisons; and so on.

[0037] However, unlike traditional single-step API call-based agents, the execution process of GUI agents is characterized by multi-step dependencies, tool call correlations, and operational logic coherence. The quality of their task completion depends not only on the final result but also on the rationality of the execution trajectory.

[0038] In view of this, this specification provides a technical solution for generating evaluation data for graphical user interface agents (hereinafter referred to as GUI agents). By pre-constructing the optimal execution path of a human expert as a reference trajectory, when generating evaluation data for the GUI agent, the execution trajectory obtained by the GUI agent executing user instructions is acquired. Then, the execution trajectory is compared with the reference trajectory, and the indicator values ​​on various evaluation metrics are extracted as evaluation features. Then, based on the mapping of indicator values ​​to the same value range, the corresponding evaluation scores for each predetermined evaluation metric are calculated independently. Evaluation data for the GUI agent is generated based on each evaluation score. The evaluation data generated in this way is traceable to each evaluation metric, possessing traceability. By comparing with the reference trajectory, the rationality of each evaluation metric can be evaluated, thus providing a comprehensive and standardized GUI agent evaluation paradigm capable of effectively evaluating various agents. The evaluation results can be used for business processing such as the selection and optimization of GUI agents, providing detailed references for related business processes.

[0039] The technical concept for generating evaluation data for graphical interface intelligent agents is described in detail below with reference to the accompanying drawings.

[0040] Figure 3 A process for generating evaluation data for a graphical user interface (GUI) agent is illustrated according to one embodiment. The agent executing this process can be any computer, device, or server with a certain computing power. The GUI agent is used to predict and execute operations performed on the user's graphical interface to complete the target task described by the user's instructions, based on the user's graphical interface and user commands.

[0041] like Figure 3 As shown, the process of generating evaluation data for a graphical user interface (GUI) agent may include the following steps: Step 301, obtaining a first execution trajectory obtained by the GUI agent executing a first user instruction, wherein the first user instruction corresponds to a first task type; Step 302, based on a comparison with a pre-constructed first reference trajectory, extracting first evaluation features from the first execution trajectory for each predetermined evaluation indicator, wherein the first evaluation features include indicator values ​​on each predetermined evaluation indicator, and the first reference trajectory is obtained based on the first task type; Step 303, mapping each indicator value in the first evaluation features to the same value domain to obtain each evaluation score corresponding to each predetermined evaluation indicator; Step 304, generating first evaluation data based on each evaluation score.

[0042] The following details each step.

[0043] First, in step 301, the first execution trajectory obtained by the GUI agent executing the first user instruction is acquired.

[0044] In this specification, user instructions can be instructions used to describe a target task that a user expects to complete based on the current user interface. The first user instruction can be a user instruction that invokes any task of the GUI agent. For example... Figure 2 In the example shown, the first user instruction could be "take a taxi to ** (location)," and the target task is to complete the operation of "taking a taxi to **." The target task described by the first user instruction can be denoted as the first target task. The first user instruction can be any form of data, such as text or audio, and is not limited here.

[0045] It's understandable that to complete a specific task, a series of operations might be performed on the user's graphical interface. The GUI agent can predict and execute these operations, which together form an execution trajectory. This trajectory can include multiple operations, each corresponding to an operation type and an operation element. The operation type describes the action to be performed, such as clicking, inputting, or swiping. The operation element refers to the element on the page where the operation is performed, such as a button, input box, or scroll wheel. For example, an operation might be "Click, Confirm Button," where "Click" is the operation type and "Confirm Button" is the operation element.

[0046] The execution trajectory corresponding to the first user instruction can be recorded as the first execution trajectory. The first execution trajectory can be obtained based on device operation logs, operation logs, and keyboard and mouse records (keyboard and mouse operations). The first execution trajectory may include multiple atomic operation types (i.e., the smallest operation type) and operation elements (i.e., the target elements of atomic operations). A single-step operation is denoted as Step. i The first execution trajectory with k steps can include an operation sequence. .

[0047] In some embodiments, the acquired initial trajectory data can also be cleaned. This includes removing invalid operations, standardizing operation elements, etc. Invalid operations include, for example, operations that are unresponsive to the system, repeated waiting, etc. Operation element standardization can be achieved through a unified description method.

[0048] In an optional embodiment, the execution trajectory may further include the task type. Task Description Operation results Number of operation steps Operation path, execution time At least one of the following: The task type describes the task type corresponding to the user instruction, such as sending an email, hailing a taxi, modifying a file, etc. The task type corresponding to the first user instruction and the first execution trajectory is the first task type; the task description is used to describe the task to be performed in natural language; the operation result can be the execution result, such as page redirection, etc.; the number of operation steps is the number of atomic operations included; the operation path can be the path of one or more atomic operations combined to achieve a sub-goal through natural language description, such as "open file - modify data - save file," etc., which is different from the execution trajectory described by atomic operations; the execution time can be determined based on the time from receiving the instruction to the completion of the task, such as the time difference obtained by subtracting the system response time from the time from receiving the instruction to the completion of the task. For example, a specific cleaned trajectory data can be recorded as:

[0049] .

[0050] As a specific example, if the first user instruction is "Send an email with an attachment", and the current page is the system desktop, then the task description could be "Use the GUI agent on the desktop to send an email with an attachment". The resulting operation path could be: "Open the email on the desktop, add an attachment, and send the email". The execution trajectory could be: "[Click, browser icon; enter email login page, search box; enter account, account input box; enter password, password input box; click, login button; click, compose button; enter recipient, recipient input box; click, add attachment button; enter, file path; click, add button; click, send button]", with 11 operation steps and an execution time of 1.5 seconds.

[0051] In a further optional embodiment, each operation can be standardized to a predetermined format, and standardization processing can be performed on the operation elements to map the operation elements to standard elements in the task benchmark library. For example, "save button in the upper right corner" can be standardized to "save button", etc. As a specific example, a single-step operation Step i The structured definition is as follows: Step i ={Idx i OpType i TargetElem i , Loc i Content i Tool i , Result i Thought i IsSuccess i}. Where: Idx i Indicates the index of the operation step (e.g., 1, 2, 3, etc.); OpType iIndicates the type of atomic operation (such as click / input / scroll, etc.); TargetElem i Represents an action element (such as an OK button, send button, etc.); Loc i Represents the coordinates (x, y) of the element being operated on; Content i This represents the content of input operations. Non-input operations, such as those using predefined characters like Null, can be represented. Tool i Indicates the name of the tool being called; Result i Indicates the result of an operation (such as redirecting to a page); Thought i Indicates the agent's thought process (if any); IsSuccess i Indicates whether the operation was successful (using a Boolean value, such as 0 for failure and 1 for success).

[0052] In other embodiments, the first execution trajectory may also undergo other normalization processes, which will not be elaborated here.

[0053] Then, via step 302, a first evaluation feature is extracted from the first execution trajectory based on a comparison with a pre-constructed first reference trajectory.

[0054] The reference trajectory can be a dataset consisting of pre-constructed optimal operation sequences. The reference trajectory can be an optimal trajectory, such as the individual operation steps performed by a human expert. For example, it may include a set of key steps, that is, defining the core operation sequence to complete the task, denoted as S. key ={s1, s2, ..., s n}, where s i An atomic operation is a combination of an atomic operation and a target element. An atomic operation can be a specific action, such as a single operation type and a corresponding operation object, like "click-save button", where click indicates the operation type is "click" and the save button is the operation object.

[0055] Each reference trajectory can correspond to metadata used to describe the reference trajectory. In some embodiments, the metadata may include the task type, such as as described above. Identifiers (e.g., 1 for data processing, 2 for email sending, etc.). Thus, a reference trajectory consistent with the task type of the first user instruction can be used for comparison; this is denoted as the first reference trajectory. The task type corresponding to the first reference trajectory might be data processing, email sending, ride-hailing, etc. Typically, the task type of a reference trajectory can be selected from several task types to be evaluated; the task type corresponding to the first reference trajectory is the first task type.

[0056] According to some embodiments, the metadata of the reference trajectory may further include at least one of the following labeled data: task description, scenario category, target information, etc. The task description describes the task to be performed, such as "what to do" and "how to do it." The scenario category distinguishes different task scenarios and can be a higher-level concept of the task type, such as document operation, application operation (e.g., hailing a ride, posting on Weibo), communication operation (e.g., making a phone call), query (e.g., checking the weather), etc. The target information can be a description of the operation result, describing "what was done" and "what was achieved."

[0057] As a specific example, the task description corresponding to the first reference trajectory is as follows: "Open the file 'Q2 Sales Data.xlsx' on the desktop, find the pivot table in the 'East China Region' worksheet, and discover that the total Q2 amount for 'Shanghai Branch' is displayed as ¥3,200, but should actually be ¥32,000 (because cell C47 in the original data is missing a zero). Correct the value of cell C47, refresh the pivot table, confirm that the total Q2 amount for 'Shanghai Branch' is updated to ¥32,000, then add the comment 'Corrected on 6 / 11' to the remarks column, save the file as 'Q2 Sales Data_Corrected.xlsx' to the desktop, and do not save the changes when closing the original file." The scenario category is "Document Operation," and the target information is: "1. Correct the value of cell C47 to 32000; 2. Refresh the total Q2 amount for 'Shanghai Branch' in the pivot table to ¥32,000; 3. The new file 'Q2 Sales Data_Corrected.xlsx' exists on the desktop."

[0058] In some embodiments, a single reference trajectory may also correspond to optimal trajectory parameters obtained through statistics, such as the optimal number of steps N. opt Optimal time T opt Optimal operation path set P opt ,etc.

[0059] Wherein, the optimal number of steps N optThis represents the minimum number of steps a human expert must execute, or the minimum number of operations required to achieve the goal. The optimal number of steps can be set by a human expert or determined by statistically analyzing the number of atomic operations in a reference trajectory. For example, in the previous example, the optimal number of steps is 11, meaning the optimal operation path has 11 steps, such as: 1) Double-click the desktop file icon "Q2 sales data.xlsx"; 2) Delete ¥3,200 in cell C47; 3) Enter ¥32,000 in cell C47; 4) Enter "Corrected on 6 / 11" in the remarks column; 5) Click the "Save As" button; 6) Enter "Q2 sales data_corrected.xlsx" in the file name input box; 7) Click the "Save" button; 8) Enter the desktop as the save location; 9) Click the "OK" button; 10) Click the "Close Original File" button; 11) Click the "Don't Save Changes" button.

[0060] Optimal time T opt This can represent the shortest execution time for a human expert. The shortest execution time for a human expert can include the time spent on human actions and reactions, such as the time from the start of execution to obtaining the final result, or the task completion time minus the system response time. System response time, for example, is the time from receiving a user action to receiving system feedback, such as clicking the "OK" button to successful saving; it is the system time spent on the action "click the OK button." The total execution time from start to finish and the system response time can be obtained from the device's operation logs.

[0061] Optimal operation path P opt This represents the shortest or simplest path to complete a task. It's usually the set of steps requiring the fewest operations to achieve the goal, or the set of steps consuming the least computational resources. For example, for the task of opening Notepad, the path "Start Menu Search → Open Notepad" is superior to "Start Menu → Attachments → Notepad" and can be considered the optimal path. In the previous example, the path could be something like "1. Correct the value of cell C47 to 32000; 2. Update the total amount of Q2 in the 'Shanghai Branch' pivot table to ¥32,000; 3. Create a new file 'Q2 Sales Data_Corrected.xlsx' on the desktop.", which differs from the 11 steps 1) to 11) listed earlier.

[0062] In some optional examples, the first reference trajectory may also correspond to a list of all valid operation elements involved in the current task. Valid operation elements can be elements capable of performing valid operations, i.e., legal elements corresponding to operations, such as the operation element "Notepad editing area" for input operations, or the operation element "Save button" or "Desktop path selection box" for click operations, etc. A first reference trajectory including m valid operation elements can correspond to a list of valid operation elements, such as E. valid= {e1, e2, ..., e m In the execution trajectory, valid operations performed on valid operation elements can be recorded as valid steps.

[0063] It's worth noting that critical steps (or critical operations, which involve operating on key operation elements) emphasize the necessity of task completion—the absence of a critical step means the task cannot be completed correctly; effective steps (or effective operations, corresponding to effective operation elements) emphasize the legitimacy of the operation objective—whether the operation applies to effective operation elements relevant to the task. A step in the execution trajectory can be both a "critical step" and a "effective step," but the two are not equivalent: a effective step is not necessarily a critical step (for example, clicking on an effective element but not required by the task), and the task will still fail even if all other steps are effective steps when a critical step is missing.

[0064] In other embodiments, the reference trajectory may also correspond to other preset types of data, which will not be described in detail here.

[0065] Evaluation features can be the index values ​​of the execution trajectory on multiple predetermined evaluation metrics. The evaluation feature of the first execution trajectory can be denoted as the first evaluation feature. The first evaluation feature can be determined by comparing the first reference trajectory and the first execution trajectory, describing the difference between the first execution trajectory and the first reference trajectory.

[0066] In some possible scenarios, multiple predetermined evaluation metrics may include at least one of the following: logical jump count, element recognition accuracy, and anomaly handling rate.

[0067] Here, the logical jump count represents the number of operations in the first execution trajectory that deviate from the optimal operation path corresponding to the first reference trajectory. Specifically, a logical jump step is a step where the previous step was consistent with the optimal path, but the current step is inconsistent with the optimal path. Logical jumps often lead to incoherent step logic, and the focus is on the timing / sequentiality of deviation from the optimal path. For example, if the optimal path corresponding to the first reference trajectory is A→B→C→D→E, and the path of the first execution trajectory is A→B→D→C→E, then the step from B to D (B→D) is a jump (skipping C). D and C are both valid steps, and the number of steps is not much, but the logical order is incorrect; this is a logical jump. Logical jump count = |{S} tepi |S tepi ∉P opt ∧S tepi−1 ∈P opt}∣。 P opt Step represents the optimal operation path in the first reference trajectory. i Step represents the i-th step (i-th operation) in the first execution trajectory. i-1 This is the (i-1)th step in the first execution trajectory.

[0068] Element recognition accuracy describes the precision of element identification. Typically, an atomic operation includes an operation type and an operation element. A correct step is defined as the step where the operation element is correctly identified and its coordinates are correct within the first execution trajectory. In practice, the GUI agent can locate the operation element and perform operations according to its coordinates. An operation element often corresponds to an operation range. For example, if operation step A is "click the save button," then the operation element "save button" can be operated within the coordinate range [a±5, b±5] on the interface. If the operation element corresponding to the first execution trajectory is correct and its coordinates are within this range (correct operation coordinates), then the element is accurately identified and corresponds to the correct step. The ratio of the number of correct steps to the total number of steps in the first execution trajectory can be used as the element recognition accuracy.

[0069] The exception handling rate measures the GUI agent's ability to handle exceptions. For example, it's the ratio of the number of corrective steps in a corrective operation to the number of exceptions in the first execution trajectory. Exceptions can be steps such as operation failures or unresponsive actions, such as empty clicks, incorrect positioning, etc. As a concrete example, the exception handling rate can be expressed by the following formula:

[0070]

[0071] Where i represents the i-th step in the first execution trajectory, IsSuccess i This indicates whether the i-th step was successful. This indicates an unsuccessful or abnormal operation, represented by the number 0. Indicates AND and intersection. Indicates abnormal operation steps The next step is to correct the abnormal operation, where || represents the quantity statistics in this paper.

[0072] In optional embodiments, the predetermined evaluation metric may further include an error operation rate, used to measure the legitimacy of the GUI agent's operations. This rate can be measured by the frequency of generating invalid operations (such as operations not included in the valid operations corresponding to the first reference trajectory). The error operation rate can be expressed as the ratio of invalid operations to all operations, such as: Error operation rate = For example, the effective operation E in the optimal path A→B→C→D→E corresponding to the first reference trajectory. valid There are 5 options, A, B, C, D, and E. The path corresponding to the first execution trajectory is A→B→X→D→E. Therefore, operation X is not in E. validThis is an erroneous operation. The erroneous operation rate is the ratio of erroneous operand 1 to the total operands 5. It is worth noting that jumping from step B to step X also constitutes a logical jump (the number of logical jumps can be calculated, and it is independent of the erroneous operation rate), which will not be elaborated on here.

[0073] In an optional embodiment, the predetermined evaluation metric may further include the number of missing critical steps, used to measure the completeness of critical steps of the GUI agent. Critical steps are typically essential operational steps for achieving the task objective, and are denoted as critical steps. For the first execution trajectory, the steps it contains can be matched with the critical steps of the first reference trajectory. Steps in the critical step set that are not in the first execution trajectory are counted as missing critical steps. For example, denoted as... .in, This represents a set of critical steps, where each step corresponds to an operation type and a target operation element. S traj ={OpType i +TargetElem i |Step i ∈Traj} represents the operation type and target operation element of the i-th step in the path of the first execution trajectory. If both the operation type and target operation element of the first execution trajectory are the same, it indicates that the steps are identical. Taking saving a file as an example, key steps may include: clicking the save button → selecting the save path → entering the filename → clicking the OK button. If the steps of selecting the save path and entering the filename are missing, the file will be saved in the default path and may not be named. In this case, the number of missing key steps is 2.

[0074] In an optional embodiment, the predetermined evaluation metric may further include the number of erroneous tool calls, used to verify the correctness of tool calls by the GUI agent. In this case, each step in the first reference trajectory may also correspond to a valid toolset, used to describe whether each step correctly calls a tool. A single step may correspond to one or more valid operation tools. If a step is correct, it is checked whether the called tool is the corresponding valid operation tool. This can be denoted as: Number of erroneous tool calls = |{Step...} i |Tool i ∉Tool valid (OpType i )}∣, where Step i Tool represents step i in the first execution trajectory. valid (OpType i ) indicates the corresponding atomic operation type (OpType) i Effective operating tools under ) Tool iThis indicates the tool actually invoked in step i of the first execution trajectory, with ∉ indicating it does not belong to the set. For example, if the click operation corresponds to the tool system_gui_click, and a click operation is performed in step i, but the tool system_gui_input is actually used in a certain click step of the first execution trajectory, resulting in no response to the click, the error tool call count is incremented by 1.

[0075] In an optional embodiment, the predetermined evaluation metric may further include redundancy rate, used to evaluate the redundancy of instructions executed by the GUI agent. The redundancy rate can be the number of steps N executed by the GUI agent in the first execution trajectory. total Optimal number of steps corresponding to the first reference trajectory The difference between them, and The ratio, denoted as: If the optimal number of steps is 5, the number of steps in the first execution trajectory is 5, the redundancy rate is 0.0, and there are no meaningless repeated operations. If, after a certain operation (such as opening Notepad), a blank area is clicked repeatedly 8 times, the total number of steps will be 13, and the redundancy rate will be 1.6.

[0076] In an optional embodiment, the predetermined evaluation metric may further include a time consumption ratio, used to assess the execution time performance of the GUI agent. The time consumption ratio may be the execution time of the first execution trajectory compared to the execution time corresponding to the first reference trajectory. That is, the optimal time T opt The ratio, execution time can be the total time T from receiving the instruction to the end of execution. total It can also be the total time T. total With system response time T sys The time difference. For example, if the execution time is the time difference, then: .

[0077] In optional embodiments, the predetermined evaluation metric may further include path matching degree, used to detect the quality of the GUI agent's path planning ability. Path matching degree is typically compared to the optimal path, and is defined, for example, as: That is, the optimal path in the first execution trajectory compared to the first reference trajectory. Consistent number of steps (satisfying) (Number of steps), compared with the total number of steps in the first execution trajectory. The ratio of .

[0078] In optional embodiments, the predetermined evaluation metric may further include environmental adaptability, where "environment" can be understood as the environment in which the operation is performed, such as the coordinate range of the click, the coordinates of the input box, etc. For example, when clicking the "Save" button, the click position can be determined based on the "Save" button on the interface. Environmental adaptability means that the click position is correct. The click position can be a single coordinate or a range of coordinates. To cope with interface offsets, when the click position is a single coordinate, the adaptation range can be expanded up, down, left, and right of the click point, such as expanding by 20 pixels in each direction. When the click position is a range of coordinates, the adaptation range is the range of coordinates itself, or it can be a larger range of coordinates that expands to a reasonable position around it. (Using Loc...) i This represents the coordinates of the operation element at step i in the first execution trajectory. This indicates the coordinate adaptation range of the valid operand e, after operand e is replaced with the valid operand e. In the case of: .

[0079] In an optional embodiment, the predetermined evaluation metric may further include the number of correction steps. The number of correction steps refers to the number of corrective operation steps for the erroneous step, such as deleting the incorrect input, clicking again, modifying the operation type of the atomic operation, etc. The correction steps may carry a correction identifier, such as the step being "corrected to:...". For example, if the atomic operation of step i is "input", and there is no input field on the current page, the operation fails (e.g., IsSuccess). i If the first execution path is i+1 (i=0), and the next step is a correction to the operation (e.g., clicking the "selection box button for option C"), then the correction step count is incremented by 1. The correction step count is usually evaluated along with the error step count to assess GUI performance. For example, if you enter the filename "eval.txc" (an input error), directly deleting the trailing character "c" and entering "t" completes the correction in one step, which is a relatively successful correction. However, repeatedly deleting and entering the file multiple times (e.g., four times) without success, or saving without performing any correction after discovering the error, are all unsuccessful corrections. Therefore, the correction step count can be statistically correlated with the error step count. For example, in the first execution path, the correction step count is the average of the correction steps for each error step.

[0080] In optional embodiments, the predetermined evaluation metric may further include the number of irrelevant resources used to assess the rationality of the GUI agent's resource usage. Resource usage can be measured by the number of processes. Irrelevant resources are resources other than the necessary resources for executing the first user instruction. Necessary resources are those occupied by processes that cannot be bypassed. For example, if the first user instruction is "record meeting content in Notepad", the necessary resource is the resource occupied by the "Notepad.exe" process. If the first execution trajectory also includes at least one irrelevant process such as opening a browser, calculator, image viewer, or media player, then the number of irrelevant resources used is increased by the corresponding number (e.g., 1, 2, etc.). Necessary resources can be determined based on the first reference trajectory, for example, the first reference trajectory corresponds to the effective process set Process. valid Through matching, during the process of the GUI agent obtaining the first execution trajectory, one of the opened processes did not belong to the valid process set. valid Then, the irrelevant resource usage count is incremented by 1, and the irrelevant resource usage count = |{Process i |Process i ∉Process valid}∣.

[0081] In other embodiments, the predetermined evaluation indicators may also include other evaluation indicators, which will not be elaborated here. Using multiple predetermined evaluation indicators can provide a more comprehensive evaluation system for the GUI agent.

[0082] Next, in step 303, the values ​​of each indicator in the first evaluation feature are mapped to the same value range to obtain the evaluation scores corresponding to each predetermined evaluation indicator.

[0083] The first evaluation feature reflects the specific performance indicators of the GUI agent in processing the first user command from different perspectives. In the evaluation of the GUI agent, it is also desirable to more intuitively quantify the level to which each specific performance indicator belongs. Therefore, the first evaluation feature can be mapped to the respective evaluation scores for each predetermined evaluation indicator.

[0084] Mapping the first evaluation feature to the corresponding evaluation scores for each predetermined evaluation indicator aims to measure various indicators under the same scoring criterion. For example, each evaluation score might correspond to the same value range [0,1], or to the same level of numerical value (0, 1, 2, 3), etc. In other words, the evaluation scores for each predetermined evaluation indicator correspond to the same value range. The mapping from the first evaluation feature to each evaluation score can be based on a pre-trained neural network or on predetermined rules.

[0085] In the case of using pre-trained neural networks, the number of neural networks can be the same as the number of predetermined evaluation metrics. The output of a single neural network is a single numerical value, i.e., the evaluation score of the corresponding evaluation metric. The neural networks used can be of various reasonable architectures, such as fully connected neural networks. It is worth noting that during neural network training, training samples can correspond to manually labeled evaluation scores. The goal of training the neural network is to minimize the difference between the output of the neural network and the manually labeled evaluation scores. Each neural network can map each dimension of the first evaluation feature to evaluation scores within the same value range.

[0086] When based on predetermined rules, various rules conforming to natural laws can be pre-set to describe the correspondence between indicator values ​​and scores. For example, referencing the concept of a rubric, the corresponding mapped scores are obtained based on the magnitude conditions (usually described by value ranges) that the indicator values ​​corresponding to each evaluation indicator must satisfy. In this case, the mapped scores are typically multiple preset level values. Table 1 below provides an example of determining evaluation scores using predetermined rules.

[0087] Table 1. Examples of evaluation score determination based on predetermined rules

[0088] Operational effectiveness Error rate 5 points: Error rate = 0; 3 points: 0 < Error rate ≤ 0.2; 0 points: Error rate > 0.5 Completeness of key steps Missing critical steps 5 points: 0 missing critical steps; 2 points: 1 missing critical step; 0 points: ≥2 missing critical steps Correctness of tool invocation Error tool call count 5 points: Error tool call count = 0; 1 point: Error tool call count ≥ 1 and FinalResult (final execution result) = Success; 0 points: Error tool call count ≥ 1 and FinalResult = Failure. Logical coherence of steps Logical jump number 5 points: Logical jump count = 0; 2 points: 0 < logical jump count ≤ 2; 0 points: Logical jump count > 2 or logical reversal. Reasonableness of target element identification Element recognition accuracy 5 points: Element recognition accuracy = 1; 2 points: 0.8 ≤ Element recognition accuracy < 1; 0 points: Element recognition accuracy < 0.5 Reasonableness of exception handling Anomaly handling rate 5 points: Exception handling rate = 1; 1 point: 0 < exception handling rate < 1 and FinalResult = Success; 0 points: Exception handling rate = 0 Step redundancy Redundancy 5 points: Redundancy rate ≤ 0.2; 3 points: 0.2 < redundancy rate ≤ 0.3; 0 points: Redundancy rate > 0.5 Reasonableness of execution time Time consumption ratio 5 points: Time spent ≤ 1.5; 2 points: 1.5 < Time spent ≤ 2; 0 points: Time spent > 3 Path optimality Path matching degree 5 points: Path matching degree = 1; 2 points: 0.7 ≤ Path matching degree < 1; 0 points: Path matching degree < 0.5 Environmental adaptability Environmental adaptability 5 points: Fit rate = 1; 3 points: 0.8 ≤ Fit rate < 1; 0 points: Fit rate < 0.5 Error recovery capability Correction Steps 5 points: Correction steps ≤ 1; 2 points: 1 < Correction steps ≤ 3; 0 points: Correction steps > 3 or no correction. Rationality of resource utilization irrelevant resource usage 5 points: Irrelevant resource usage = 0; 3 points: 0 < irrelevant resource usage ≤ 2; 0 points: irrelevant resource usage > 5

[0089] In practice, other predefined rule settings can also be used, which will not be elaborated here. To explain, logical reversal means that the two steps are executed in reverse order.

[0090] Each evaluation score reflects the performance of the GUI agent across different dimensions. Through mapping, the feature values ​​of each dimension are placed in the same feature space for evaluation. More specifically, the same value in each evaluation score indicates that the performance of the corresponding different dimensions is at the same level.

[0091] Further, in step 304, first evaluation data is generated based on each evaluation score.

[0092] Based on the various evaluation scores, evaluation data for the agent can be generated, denoted as the first evaluation data. The first evaluation data can be evaluation data generated solely for the first execution trajectory, or it can be evaluation data generated for multiple execution trajectories, including the first execution trajectory; there is no limitation here. The evaluation data can include at least one of the following: evaluation results for each evaluation metric dimension, and overall evaluation results.

[0093] In one embodiment, the evaluation results can be obtained based on predetermined generation rules. The evaluation results for each evaluation metric dimension can be the evaluation results from each dimension according to the predetermined generation rules. The evaluation score for a single dimension corresponds to a performance level; the higher the score, the higher the performance equivalence, and the lower the score, the lower the performance level. For example, an evaluation score of 5 indicates that the GUI agent performs excellently in the corresponding evaluation metric dimension, while an evaluation score of 0 indicates that the GUI agent performs poorly in the corresponding evaluation metric dimension.

[0094] The overall evaluation result can include the fusion of evaluation results from various evaluation indicator dimensions. In one optional example, the evaluation results for each dimension can be summarized, for example: good performance in evaluation indicators A, B, and C, moderate performance in evaluation indicators D and E, etc. In another optional example, the overall evaluation result can also be determined based on the fused score obtained by fusing the individual evaluation scores. The fused score can be the sum, average, or weighted sum of the individual evaluation scores (with weights set in advance), etc.

[0095] In a specific example, the evaluation indicators can be categorized, with each category corresponding to a preset weight. First, the scores within each category are summed or averaged to obtain a category evaluation score. Then, the preset weights are used to weight the evaluation scores of each category. For example, the evaluation indicators can be divided into four categories: correctness, rationality, efficiency, and robustness. Specifically, taking the evaluation indicators shown in Table 1 as an example, the correctness indicator category can correspond to evaluation indicators such as error operation rate, number of missing key steps, and number of incorrect tool calls; the rationality indicator category can correspond to evaluation indicators such as number of logical jumps, element recognition accuracy, and anomaly handling rate; the efficiency indicator category can correspond to evaluation indicators such as redundancy rate, time consumption ratio, and path matching degree; and the robustness indicator category can correspond to evaluation indicators such as environment adaptability rate, number of correction steps, and number of irrelevant resource occupancy. The above classification is only an example; other classification methods can be used in other embodiments.

[0096] Weights are assigned based on the importance of each indicator category (weights can be set manually based on experience, or determined in other ways, such as through the Analytic Hierarchy Process (AHP)). The weight vector is denoted as W = [w1, w2, w3, w4]. The average evaluation score for the j-th indicator category is: (j=1, 2, 3, 4, representing the indicator category; n) j (Number of evaluation indicators). The fusion score is: The overall evaluation result can be determined based on the fusion score.

[0097] When determining weights using the Analytic Hierarchy Process (AHP), the following steps are performed: Construct a hierarchical model, such as dividing the evaluation system into three layers: the target layer is the "GUI agent comprehensive evaluation score," the criteria layer consists of various indicator categories (e.g., correctness C1, rationality C2, efficiency C, robustness C4), and the solution layer consists of specific evaluation indicators under each indicator category (e.g., error rate, logical jump count, etc.); Construct a judgment matrix, inviting multiple (e.g., 3-5) experts in the field of GUI agent evaluation to perform pairwise comparisons of each indicator category in the criteria layer using a predetermined range scaling method (e.g., 1-9, 0-5, etc.), constructing a judgment matrix A (rows and columns correspond to each indicator category). For example, if experts believe that "correctness" is significantly more important than "rationality," then the element 'a' in the first row and second column of the matrix... 12 =3, element a in the second row and first column 21 =1 / 3, matrix elements satisfy aᵢⱼ>0, aⱼ i =1 / aᵢ j 1. Calculate the weight vector and perform a consistency test. Normalize the judgment matrix A by columns and calculate the row average to obtain the initial weight vector W. If there are 4 index categories, W = [w1, w2, w3, w4]ᵀ. Calculate the largest eigenvalue λ of the judgment matrix A. max Then, the consistency index C is obtained. I =(λ max -n) / (n-1), where n is the matrix order. The random consistency index R is obtained by looking up the table (the average random consistency index table, corresponding to n). I (When n=4, R) I =0.90), calculate the consistency ratio CR=C I / R I If CR < 0.1, the judgment matrix can be considered to have satisfactory consistency, and the weight vector W is valid; otherwise, the judgment matrix needs to be adjusted and recalculated.

[0098] When the initial evaluation data comprises evaluation data generated from multiple execution trajectories, including the first execution trajectory, the evaluation scores corresponding to each execution trajectory can be combined using methods such as summation, averaging, or statistical calculation. Taking statistical calculation as an example, performance on the corresponding evaluation metric can be determined based on the proportion of each value. For instance, for 100 execution trajectories corresponding to 100 samples, in the resource utilization rationality dimension, among the evaluation scores for the irrelevant resource utilization number, 80 are 5 points, 15 are 1 point, and 5 are 0 points. The proportion of 5 points is significantly higher than (e.g., higher than a predetermined threshold) than the number of other scores, then the performance in the irrelevant resource utilization number can be rated as excellent, and so on.

[0099] In other embodiments, a large language model can be used to assist in generating the first evaluation data. In this case, the various evaluation scores and the task description information for generating the evaluation data can be used as prompts for the large language model to generate the evaluation data. For example, the task description information describes the generated task as follows: output a structured evaluation report containing "total score, scores for each dimension, problem location, and optimization suggestions," etc.

[0100] In an optional embodiment, for each evaluation score that meets predetermined conditions, manual review can be performed before it is used to generate evaluation data. Manual review can be performed on lower-scoring dimensions (such as the average score D corresponding to the indicator category). j < 3) Or boundary cases, where human experts correct scoring biases based on experience and mark specific problems. Boundary cases can be one of the following: extreme values ​​in the execution trajectory, such as all 5s or all 0s; abnormal evaluation metrics after using alternative solutions, meaning that when evaluation metrics are abnormal, an alternative solution is checked and used if determined. A specific example of using an alternative solution is: the task "Save file to desktop," with the key steps set as {click save → select desktop → enter filename → confirm}. The first execution trajectory does not execute the key step "select desktop" (missing one step), but instead directly enters `C:\Users\xxx\Desktop\file.txt` in the file path field, and the file is indeed saved to the desktop. Evaluation scores may be biased for alternative solutions: for example, if one key step is missing, the evaluation score for the number of missing key steps is determined to be 2 points according to predetermined rules. However, the GUI agent actually completes the same goal through the alternative solution, and the score may be adjusted to 3 to 4 points through human review.

[0101] In some embodiments, to determine whether an alternative is an option, at least one of the following methods can be used for automatic determination: equivalence mapping table determination, where the reference trajectory library can be expanded to cover equivalent cases under the same task type, and whether an alternative is an option is determined by matching the steps in the first execution trajectory with various reference trajectories; semantic similarity determination, where whether an alternative is an option can be determined based on semantic fuzzy matching between the first execution trajectory and the first reference trajectory. Semantic fuzzy matching can, for example, be based on a semantic embedding model to obtain the embedding vectors of the first execution trajectory and the first reference trajectory, and then calculate the embedding similarity.

[0102] The goal of evaluating GUI agents is typically to select superior agents or to optimize them for better performance. When optimizing GUI agents, various defects in the execution trajectory can be identified, allowing for targeted iterative optimization. For example, based on evaluation data, weak dimensions of the GUI agent (such as element recognition and logical coherence) can be determined. In these dimensions, the weight of the loss function can be increased, or the number of high-quality samples can be increased, thereby optimizing the GUI agent by minimizing model loss.

[0103] Reviewing the above process, the technical solution for generating evaluation data for graphical user interface (GUI) agents provided in this specification compares the execution trajectory obtained by the GUI agent executing user commands with a pre-constructed optimal reference trajectory to extract corresponding evaluation features. These features include indicator values ​​for various predetermined evaluation metrics. Mapping each indicator value in the evaluation features to the same value domain yields corresponding evaluation scores for each predetermined evaluation metric. Based on these scores, traceable evaluation data can be generated. Thus, GUI agents can be evaluated from multiple dimensions, enabling comparisons between multiple GUI agents or targeted optimization of existing GUI agents. By setting comprehensive evaluation metrics and independent mapping of indicator values ​​to the same value domain, a more accurate and stable evaluation of GUI agents can be achieved through a precise and traceable evaluation system.

[0104] The technical advantages of the technical solution in this specification compared to conventional technologies are reflected in the following aspects:

[0105] Interpretability: It can locate specific problems in the trajectory (such as "missing key steps: desktop path not selected", "step redundancy: 0.3, exceeding the optimal number of steps by 30%", etc.), providing a clear direction for the iterative optimization of the GUI agent;

[0106] Standardization: Construct unified evaluation criteria so that evaluation results of different GUI agent tasks and different GUI agents can be compared horizontally, solving the problem of inconsistent evaluation standards within the industry;

[0107] High efficiency: The automated scoring process supports large-scale trajectory evaluation (tens of thousands of trajectories / hour), and only boundary cases (indicator evaluation scores that meet predetermined conditions) require manual review, which is more than 1,000 times more efficient than pure manual evaluation;

[0108] Comprehensiveness: It can cover multiple indicator categories, such as correctness, rationality, efficiency, and robustness, evaluating both "whether the task is completed" and "how the task is completed", comprehensively measuring agent performance;

[0109] Adaptability: Quickly adapt to different GUI agent tasks (such as desktop, web page, application, etc.) through configuration files. Adding a new task type only requires updating the task benchmark library, without modifying the core scoring logic.

[0110] Quantitative: All evaluation metrics are quantified independently, and the scoring results can be used for numerical comparison of GUI agent performance, trend analysis, and verification of optimization effects.

[0111] According to another embodiment, an apparatus for generating evaluation data for a graphical user interface agent is also provided. This apparatus can be located in any computer, device, or server with a certain computing power, for example... Figure 2 The server-side or user-side terminal of map-related applications. Figure 4 An apparatus 400 for generating evaluation data for a graphical user interface agent according to one embodiment of this specification is shown.

[0112] like Figure 4 As shown, the apparatus 400 for generating evaluation data for a graphical user interface agent may include: an acquisition unit 401 configured to acquire a first execution trajectory obtained by the graphical user interface agent executing a first user instruction, wherein the first user instruction corresponds to a first task type; an extraction unit 402 configured to extract a first evaluation feature from the first execution trajectory based on a comparison with a pre-constructed first reference trajectory for multiple predetermined evaluation indicators, wherein the first evaluation feature includes indicator values ​​on each predetermined evaluation indicator, and the first reference trajectory is acquired based on the first task type; a mapping unit 403 configured to map each indicator value in the first evaluation feature to the same value domain to obtain each evaluation score corresponding to each predetermined evaluation indicator; and an evaluation unit 404 configured to generate first evaluation data based on each evaluation score.

[0113] It is worth noting that, Figure 4 The device 400 shown and Figure 3 Corresponding to the described method, Figure 3 The corresponding descriptions in the illustrated method embodiments also apply to apparatus 400, and will not be repeated here.

[0114] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 The methods described above.

[0115] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 The methods described above.

[0116] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0117] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.

Claims

1. A method for generating evaluation data for a graphical user interface intelligent agent, the method comprising: Obtain the first execution trajectory obtained by the graphical interface intelligent agent executing the first user instruction, wherein the first user instruction corresponds to the first task type; Based on the comparison with a pre-built first reference trajectory, a first evaluation feature is extracted from the first execution trajectory for multiple predetermined evaluation indicators. The first evaluation feature includes the indicator value on each predetermined evaluation indicator. The first reference trajectory is obtained based on the first task type. Map each index value in the first evaluation feature to the same value range to obtain each evaluation score corresponding to each predetermined evaluation index. The first set of evaluation data is generated based on the scores of each evaluation.

2. The method as described in claim 1, wherein, First evaluation features are extracted from the first execution trajectory for multiple predetermined evaluation indicators, including at least one of the following: Determine the number of operands in the first execution trajectory that deviate from the first reference trajectory for the optimal operation path; determine the index value of the predetermined evaluation index logical jump number based on the operands; Determine the correct number of steps for the operation elements in the first execution trajectory that are correctly identified and have the correct operation coordinates; The index value for the accuracy of the predetermined evaluation index element recognition is determined based on the ratio of the number of correct steps to the total number of steps in the first execution trajectory. The number of correction steps in the first execution trajectory is determined, and the index value of the predetermined evaluation index, the anomaly handling rate, is determined based on the ratio of the number of correction steps to the number of abnormal operations in the first execution trajectory.

3. The method as described in claim 2, wherein, The plurality of predetermined evaluation indicators also include at least one of the following: The error operation rate is the ratio of the number of erroneous operations in the first execution trajectory to the total number of operations. The number of missing critical steps is the number of critical steps that are missing from the first execution trajectory relative to the critical step set determined based on the first reference trajectory. Error tool call count is the number of steps in the first execution trajectory where a valid tool did not call the corresponding operation. Redundancy rate, which is the number of steps N in the first execution trajectory. total Optimal number of steps corresponding to the first reference trajectory The difference between The ratio; The time-to-execution ratio is the execution time of the first execution trajectory compared to the optimal execution time T corresponding to the first reference trajectory. opt The ratio; Path matching degree is the optimal path in the first reference trajectory among the steps of the first execution trajectory. The number of consistent steps, compared to the total number of steps in the first execution trajectory. The ratio; Environment adaptability rate is the ratio of the number of steps in the first execution trajectory where the operation element is a valid element determined based on the first reference trajectory and the operation position is correct to the total number of steps. The correction step count is the number of correction operation steps in the first execution trajectory. The irrelevant resource usage count refers to the number of unnecessary processes opened during the determination of the first execution trajectory.

4. The method of claim 1, wherein, The first reference trajectory is constructed in the following way: For the first candidate task under the first task type, the following labeled data is determined: task description, task type, and operation result; Obtain the operation sequence of the first candidate task performed manually as the target operation sequence; The first reference trajectory is generated based on the target operation sequence.

5. The method of claim 4, wherein, The target operation sequence includes each atomic operation and each target element corresponding to each atomic operation. Generating the first reference trajectory based on the target operation sequence further includes: The trajectory parameters based on the target operation sequence statistics include at least one of the following: the optimal number of steps describing the number of atomic operations, the optimal time determined based on the task completion time, the optimal operation path describing the sub-goal achieved by merging one or more atomic operations in natural language, and the effective operation elements including each target element. The first reference trajectory is formed based on the trajectory parameters and the target operation sequence.

6. The method of claim 1, wherein, Before extracting the first evaluation feature from the first execution trajectory based on a comparison with a pre-constructed first reference trajectory, the method further includes: The first execution trajectory is subjected to at least one of the following preprocessing operations: removing invalid operations and standardizing operation elements.

7. The method of claim 1, wherein, Mapping each index value in the first evaluation feature to the same value range yields the respective evaluation scores for each predetermined evaluation index, including: For a single predetermined evaluation indicator, based on predetermined rules, the corresponding score in the same value range is assigned according to the magnitude conditions that the indicator value meets, and the predetermined rules describe the correspondence between the magnitude of the indicator value and the score.

8. The method of claim 1, wherein, Based on the various evaluation scores, the first evaluation data is generated, including: The fusion score of the first execution trajectory is determined by the weighted sum of the scores of each evaluation. Based on the individual evaluation scores and the fusion score, the large language model outputs the first evaluation data.

9. The method of claim 8, wherein, Each predetermined evaluation indicator corresponds to multiple indicator categories, and each indicator category corresponds to a weighted weight. The fusion score is determined based on the weighted sum of the various evaluation scores, including: Determine the average evaluation score for each indicator category; The fusion score is obtained by summing the average evaluation scores according to the weighted weights of each category.

10. The method of claim 1, wherein, Based on the various evaluation scores, the first evaluation data is generated, including: For each evaluation score of multiple execution trajectories, including the first execution trajectory, the scores are combined on each predetermined evaluation indicator by one of the following methods: summation, averaging, or statistical value calculation, to obtain the combined evaluation parameters. Evaluation data for the agent is generated using the various fusion evaluation parameters.

11. The method of claim 1, wherein, Mapping each index value in the first evaluation feature to the same value range yields the respective evaluation scores for each predetermined evaluation index, including: Detect whether each evaluation indicator meets predetermined conditions, the predetermined conditions include: the average score of the internal evaluation indicators of the indicator category is lower than a predetermined value, or there are extreme values, or there are abnormal evaluation indicators using alternative solutions, and the alternative solutions are determined by extended equivalence mapping or semantic similarity. For evaluation indicators that meet the predetermined conditions, the corresponding evaluation scores are determined by manual review.

12. An apparatus for generating evaluation data for a graphical user interface intelligent agent, the apparatus comprising: The acquisition unit is configured to acquire a first execution trajectory obtained by the graphical interface intelligent agent executing a first user instruction, wherein the first user instruction corresponds to a first task type; The extraction unit is configured to extract a first evaluation feature from the first execution trajectory based on a comparison with a pre-built first reference trajectory for multiple predetermined evaluation indicators. The first evaluation feature includes the indicator value on each predetermined evaluation indicator. The first reference trajectory is obtained based on the first task type. The mapping unit is configured to map each index value in the first evaluation feature to the same value range to obtain each evaluation score corresponding to each predetermined evaluation index. The evaluation unit is configured to generate the first evaluation data based on each evaluation score.

13. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-11.

Images