Method and apparatus for evaluating GUI operation task, electronic device, and medium

Abstract

The present disclosure provides a kind of evaluation method, device, electronic equipment and medium for GUI operation task, it is related to data processing field, especially it is related to artificial intelligence field.The method can include: obtaining the task execution trajectory of task execution intelligent agent to graphical user interface GUI operation task, the task execution trajectory can include interface screenshot sequence, the GUI operation task can include at least one operation to target object through graphical user interface;Determine the evaluation standard set based on the task description of the GUI operation task, the evaluation standard set can include multiple evaluation standards independent of each other;According to the interface screenshot sequence, obtain the visual evidence set corresponding to the evaluation standard set;And based on the evaluation standard set and the visual evidence set, by task evaluation intelligent agent, determine the trajectory evaluation result of the GUI operation task.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and more particularly to artificial intelligence, specifically to an evaluation method, apparatus, electronic device, computer-readable storage medium, and computer program product for GUI operation tasks. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] With the development of multimodal large language models, intelligent agents capable of performing graphical user interface (GUI) tasks within operating system environments are increasingly being applied, such as in office scenarios like document editing, spreadsheet processing, and presentation modification. To train, optimize, and supervise the behavior of these agents, it is typically necessary to evaluate the GUI tasks they perform. How to automate the evaluation of GUI tasks is a direction worthy of attention.

[0004] The methods described in this section are not necessarily methods that had been previously conceived or adopted. Unless otherwise specified, no method described in this section should be assumed to be prior art simply because it is included in this section. Similarly, unless otherwise specified, the issues mentioned in this section should not be considered to be accepted in any prior art. Summary of the Invention

[0005] This disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for evaluating GUI operation tasks.

[0006] According to one aspect of this disclosure, an evaluation method for a GUI operation task is provided, comprising: obtaining a task execution trajectory of a task execution agent for a graphical user interface (GUI) operation task, the task execution trajectory including a sequence of screenshots, the GUI operation task including performing at least one operation on a target object through a graphical user interface; determining an evaluation criterion set based on a task description of the GUI operation task, the evaluation criterion set including multiple independent evaluation criteria; obtaining a set of visual evidence corresponding to the evaluation criterion set based on the sequence of screenshots; and determining a trajectory evaluation result of the GUI operation task by a task evaluation agent based on the evaluation criterion set and the set of visual evidence.

[0007] According to another aspect of this disclosure, an evaluation apparatus for a GUI operation task is provided, comprising: a trajectory acquisition unit, configured to acquire a task execution trajectory of a task execution agent for a graphical user interface (GUI) operation task, the task execution trajectory including a sequence of interface screenshots, the GUI operation task including performing at least one operation on a target object through the graphical user interface; a standard acquisition unit, configured to determine a set of evaluation standards based on a task description of the GUI operation task, the set of evaluation standards including multiple independent evaluation standards; an evidence acquisition unit, configured to acquire a set of visual evidence corresponding to the set of evaluation standards based on the sequence of interface screenshots; and a determination unit, configured to determine a trajectory evaluation result of the GUI operation task by a task evaluation agent based on the set of evaluation standards and the set of visual evidence.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method according to one or more embodiments of this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform a method according to one or more embodiments of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements a method according to one or more embodiments of this disclosure.

[0011] According to one or more embodiments of this disclosure, the interpretability of the evaluation process and the accuracy of the evaluation results can be enhanced, thereby contributing to more accurate graphical user interface operation.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0013] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein may be implemented according to embodiments of the present disclosure is shown; Figure 2 A flowchart of an evaluation method for a GUI operation task according to an embodiment of the present disclosure is shown; Figure 3 A schematic diagram of the data flow for an evaluation method for a GUI operation task according to an embodiment of the present disclosure is shown; Figure 4 A flowchart of an evaluation method for a GUI operation task according to another embodiment of the present disclosure is shown; Figure 5 A structural block diagram of an evaluation apparatus for GUI operation tasks according to an embodiment of the present disclosure is shown; Figure 6 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0015] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0016] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0017] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0018] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0019] Figure 1A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0020] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of one or more methods according to this disclosure.

[0021] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0022] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0023] Users can use client devices 101, 102, 103, 104, 105 and / or 106 to perform one or more methods according to this disclosure. The client devices can provide an interface that allows a user of the client device to interact with it. The client devices can also output information to the user via this interface. Although Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0024] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0025] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0026] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0027] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0028] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and 106.

[0029] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0030] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0031] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system. Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0032] The following is for reference. Figure 2 This invention describes an evaluation method 200 for GUI operation tasks according to exemplary embodiments of the present disclosure.

[0033] In step S201, the task execution trajectory of the task execution agent on the graphical user interface (GUI) operation task is obtained. The task execution trajectory may include a sequence of interface screenshots, and the GUI operation task may include performing at least one operation on a target object through the graphical user interface.

[0034] In step S202, a set of evaluation criteria is determined based on the task description of the GUI operation task. The set of evaluation criteria may include multiple independent evaluation criteria.

[0035] In step S203, a set of visual evidence corresponding to the set of evaluation criteria is obtained based on the sequence of screenshots of the interface.

[0036] In step S204, based on the evaluation criteria set and the visual evidence set, the trajectory evaluation result of the GUI operation task is determined by the task evaluation agent.

[0037] The method described in the embodiments of this disclosure can enhance the interpretability of the evaluation process and the accuracy of the evaluation results, thereby helping to achieve more accurate graphical user interface operation.

[0038] This implementation achieves logical decoupling of evaluation dimensions and precise alignment of standards and evidence. It significantly reduces ambiguity and illusions in the reasoning process of multimodal large language models, thereby enhancing the interpretability of the evaluation process and the fine-grained accuracy of the results.

[0039] Those skilled in the art will understand that a task execution trajectory refers to the operation records and corresponding sequence of interface screenshots generated by a GUI agent during task execution. It is understood that the evaluation criteria set can be a set of logically independent specific judgment rules decomposed from a complex overall task description. The visual evidence set can be key visual features or state information extracted from the screenshot sequence to verify whether a specific evaluation criterion has been met.

[0040] As a non-limiting embodiment, embodiments of this disclosure can be applied to graphical user interface (GUI) agents. Exemplarily, a GUI agent may include a task execution agent, an evaluation agent, an update agent, etc., and this disclosure may include evaluating the task execution agent for subsequent updates. Exemplarily, a GUI operation task may include performing at least one operation on a target object through a graphical user interface (GUI) agent. Exemplarily, the target object may be a file, or other forms of data, information, or other operable objects. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0041] For example, obtaining a set of visual evidence corresponding to the set of evaluation criteria based on the sequence of interface screenshots may include, for each of the plurality of evaluation criteria, determining one or more target screenshots corresponding to that evaluation criterion from the sequence of interface screenshots, and extracting visual evidence related to that evaluation criterion from the determined one or more target screenshots. For example, as will be further described below, determining the trajectory evaluation result of the GUI operation task may include determining whether each of the plurality of evaluation criteria is satisfied.

[0042] In some embodiments, obtaining a set of visual evidence corresponding to the set of evaluation criteria may include: constructing a correlation matrix between the sequence of screenshots and the set of evaluation criteria; selecting a first predetermined number of screenshots with the highest correlation for each evaluation criterion in the set of evaluation criteria to form a set of key screenshots associated with that evaluation criterion; and extracting visual evidence related to each evaluation criterion based on each screenshot in the associated set of key screenshots for each evaluation criterion in the set of evaluation criteria.

[0043] In such examples, by extracting visual evidence to obtain a screenshot-standard correlation matrix and calculating the correlation to filter out redundant visual frames that are irrelevant to the current evaluation standard, the computational overhead and processing consumption of large models can be reduced, and irrelevant screenshots can be avoided from interfering with the model's attention, thereby improving the accuracy of feature extraction.

[0044] In some embodiments, visual evidence may include at least one of the following: interface area change features, text attribute change features, layout state change features, and interaction result features.

[0045] This embodiment allows evaluation to not only focus on content changes but also capture micro-level UI typography and interactive feedback, effectively preventing the omission of fine-grained visual state changes. For example, GUI operation tasks may include making changes to the graphical user interface, and corresponding visual evidence may correspond to the features of these specific changes. In some other embodiments, visual evidence may also include other dimensions of features such as audio feedback features or cursor movement features.

[0046] In some embodiments, determining the set of evaluation criteria based on the task description may include: determining at least one execution requirement for the GUI operation task based on the task description; and generating multiple evaluation criteria based on the at least one execution requirement, such that the multiple evaluation criteria satisfy mutual independence constraints. In such embodiments, mutual independence constraints may include the probability that any one of the multiple evaluation criteria is true being independent of the probability that the other evaluation criteria are true.

[0047] According to this embodiment, multiple evaluation criteria satisfying mutual independence constraints are generated based on the at least one execution requirement, enabling each evaluation criterion to verify the GUI operation task from mutually independent dimensions. In this example, since the validity of any one evaluation criterion does not directly determine the validity of the other evaluation criteria, when one evaluation criterion fails due to the failure of individual steps, it will not lead to the failure of other evaluation criteria, thus helping to reduce misjudgments caused by the interdependence of requirements from different dimensions.

[0048] It is understood that determining at least one execution requirement for a GUI operation task based on the task description may include determining at least one explicit execution requirement and at least one implicit execution requirement. Exemplarily, at least one execution requirement may include at least one execution requirement related to the execution process and at least one execution requirement related to the execution result. It is understood that the above are merely examples, and this disclosure is not limited thereto. Additionally and / or alternatively, execution requirements may be extracted based on predefined templates or rule bases, and evaluation criteria satisfying mutual independence constraints may be generated based on these execution requirements.

[0049] In some embodiments, the mutual independence constraint may include: the difference between the conditional probability of any evaluation criterion being true under the condition that other evaluation criteria are true and the probability that any evaluation criterion is true is less than a threshold.

[0050] In such an example, the mutual independence constraint between evaluation criteria is represented as the difference between the conditional probability and the corresponding probability being less than a threshold, thus providing a calculable method for determining the degree of independence between evaluation criteria. According to this embodiment, the degree of independence between the generated evaluation criteria can be measured based on specific probability values, and the generation of evaluation criteria can be adjusted accordingly, which is beneficial for obtaining evaluation criteria with good mutual independence. Those skilled in the art will understand that the threshold can be an empirical value dynamically adjusted based on the system's fault tolerance and task complexity. It is understood that the above are merely examples, and this disclosure is not limited thereto. In some other embodiments, the mutual independence constraint between evaluation criteria can also be quantified and controlled by calculating mutual information or using other statistical independence test methods.

[0051] In some embodiments, the set of evaluation criteria may include at least one explicit evaluation criterion and at least one implicit evaluation criterion. In such embodiments, at least one explicit evaluation criterion may be used to constrain that at least one operation of a GUI operation task targets a target attribute that has been successfully changed. In such embodiments, at least one implicit evaluation criterion may be used to constrain that at least one other attribute besides the target attribute has not been changed.

[0052] According to this embodiment, the evaluation criteria include both explicit constraints on whether the target attribute has been successfully changed and implicit constraints on whether other attributes remain unchanged. In this example, while verifying that the target attribute has been correctly changed, it is also possible to check whether the GUI operation task has caused unsolicited changes to other attributes, which helps to discover side effects introduced during execution, thereby obtaining a more accurate and effective GUI agent.

[0053] It is understood that determining at least one execution requirement for a GUI operation task may include determining explicit requirements and implicit constraints for the GUI operation task, wherein implicit constraints can be used to characterize side-effect prevention conditions during the execution of the GUI operation task.

[0054] Additionally and / or alternatively, implicit constraints may further include side-effect monitoring conditions for system-level states (such as memory usage and process states).

[0055] In some embodiments, the target attribute may correspond to a first object feature that is explicitly requested to be modified in the task description, and at least one other attribute may correspond to at least one second object feature that is not requested to be modified in the task description.

[0056] Those skilled in the art will understand that at least one operation may include at least one modification operation. It is understood that a GUI operation task may include changing a target attribute in a graphical user interface, and implicit constraints may include at least one other attribute besides the target attribute remaining unchanged. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0057] In some additional embodiments, the division between the first object features and the second object features can be automatically extracted and classified from the task description using natural language processing techniques.

[0058] In some embodiments, the target object may be a file object. In such embodiments, the first object feature and the second object feature may each include parameters from at least one of the following dimensions: data content dimension, font and font size dimension, paragraph and spacing dimension, and object and layout dimension.

[0059] According to this embodiment, specific dimensions such as data, font, paragraph, and layout are defined for the file object. This embodiment provides targeted inspection dimensions for complex GUI operation scenarios, improving the comprehensiveness of the evaluation of the underlying key attributes of complex objects. It allows changes to fine-grained attributes such as font, font size, and paragraph layout to be included in the evaluation, helping to reduce the omission of such changes.

[0060] It is understood that the target attribute and other attributes can be specific target parameters and specific background parameters in the file, respectively; wherein, the specific target parameter and specific background parameter can belong to at least one of the following categories: text content, text formatting, layout, and media object. When the specific target parameter and specific background parameter belong to the same category, the specific target parameter and specific background parameter can be independent and different parameters within that category. As a more specific and non-limiting example, the target attribute and other attributes can be one or more of the following: file content, text formatting (font, font size, spacing, paragraph layout, alignment), and media item formatting. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0061] Additionally and / or alternatively, file objects may also include other dimensions of characteristics, such as metadata dimensions or security permissioning dimensions. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0062] In some embodiments, determining the trajectory evaluation result of a GUI operation task may include: determining whether the corresponding evaluation criteria are met based on a set of visual evidence; and determining the trajectory evaluation result as unsatisfactory in response to determining that at least one of the multiple evaluation criteria is not met.

[0063] In such examples, a strict veto logic is used, where failure is determined as soon as any fine-grained criterion is not met, thereby improving the accuracy and reliability of automated evaluation results.

[0064] Those skilled in the art will understand that determining whether the evaluation criterion is met based on one or more identified target screenshots may include extracting visual evidence related to the evaluation criterion from the identified one or more target screenshots, and determining whether the evaluation criterion is met based on the extracted visual evidence.

[0065] In some additional embodiments, different weights can be set so that the trajectory evaluation result is determined to be partially passed when certain low-weight evaluation criteria are not met.

[0066] For example, the evaluation method may further include determining the trajectory evaluation result of the GUI operation task based on the last consecutive second predetermined number of screenshots in the interface screenshot sequence.

[0067] According to this embodiment, a fallback screenshot can be taken based on a predetermined number of consecutive screenshots at the end, which can improve the completeness of the final task status acquisition and the accuracy of the evaluation in cases where the agent operates too quickly, the interface loading is delayed, or the state of the last frame is misjudged due to the obstruction of the pop-up window.

[0068] It is understood that the last consecutive screenshots, for example, the last five screenshots, can be used. Exemplarily, this could include inputting a predetermined number of the last consecutive screenshots from a sequence into the task evaluation agent, allowing the agent to use the final state upon task completion as a reference. In such an example, the task evaluation agent could be a multimodal agent. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0069] Additionally and / or alternatively, in addition to the screenshot at the end, screenshots of key turning points during the operation can be taken as additional reference. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0070] For example, the evaluation method may further include: performing task evaluation independently N times for the same task execution trajectory by the same task evaluation agent to obtain N single evaluation results, where N can be an odd number greater than 1; and determining the trajectory evaluation result as passed in response to the fact that the number of single evaluation results indicating that the evaluation has passed among the N single evaluation results exceeds N / 2.

[0071] This implementation introduces a statistical voting mechanism, which mitigates the inherent randomness and instability of multimodal large language models in single inference, and improves the robustness and consistency of results during large-scale system evaluation.

[0072] For example, the evaluation method may further include: obtaining an object parsing evaluation result, which may be obtained by parsing and comparing an unprocessed target object before performing at least one operation with a processed target object after performing at least one operation; and determining the comprehensive evaluation result as passed in response to the determination that both the trajectory evaluation result and the object parsing evaluation result indicate that the task has passed.

[0073] In this example, dual-path verification is performed by combining object parsing, fusing visual appearance features with underlying data features to form a positively complementary dual-path verification system. This implementation allows for further integration with underlying data (such as hidden attributes) to improve the depth and accuracy of the evaluation.

[0074] Additionally and / or alternatively, different confidence scores can be assigned to the object parsing evaluation results and trajectory evaluation results to calculate the final comprehensive evaluation score.

[0075] In some embodiments, obtaining object parsing evaluation results may include: obtaining at least one object parsing evaluation criterion; obtaining object parsing statements based on the object parsing evaluation criterion, wherein the object parsing statements can be used to parse unprocessed target objects and processed target objects; running the object parsing statements in an isolated execution environment to obtain source parsing results for unprocessed target objects and post-processing parsing results for processed target objects; and determining the object parsing evaluation result based on the difference between the source parsing results and the post-processing parsing results.

[0076] According to this embodiment, running the parsing and comparing differences in an isolated environment not only ensures the security of the host environment when executing code, but also provides an objective evaluation basis based on the actual parsing.

[0077] In some embodiments, obtaining at least one object resolution evaluation criterion may include obtaining a subset of the evaluation criteria set that is relevant to the execution result.

[0078] This embodiment achieves alignment of verification logic between multimodal visual evaluation and code parsing evaluation by using a subset of standards related to the execution results for parsing. In this embodiment, the result evaluation criteria can be a subset of all evaluation criteria.

[0079] In some embodiments, the evaluation method for GUI operation tasks described above can be implemented as part of an overall evaluation framework. Now refer to Figure 3 An operating system task automatic evaluation framework according to embodiments of the present disclosure is described, wherein the trajectory evaluation described above may correspond to the multimodal evaluation criteria evaluation path in the framework.

[0080] As an exemplary embodiment, such as Figure 3 As shown, the operating system task automatic evaluation framework (which may be exemplarily called UV Code Verifier) ​​can be used to automatically evaluate the tasks performed by GUI agents in the operating system environment.

[0081] like Figure 3 As shown in part (a), during the execution of a GUI operation task, the task execution trajectory, which includes behavior history and observation screenshots, as well as the target object before and after the operation, can be obtained.

[0082] For example, the trajectory of a single task execution can be represented as T={(o1,a1),(o2,a2),…,(o n ,a n )}, where o t It can represent the observed state of the interface at time step t (e.g., screenshot, interface elements, and system state), a t It can represent the action performed by the agent at that time step (such as clicking, typing, dragging, and keyboard shortcuts).

[0083] For example, the set of input files for a task can be represented as F. origin ={f1,f2,…,f m The set of output files after the task is completed can be represented as F. final ={f1′,f2′,…,f m ′}.

[0084] As a non-limiting example, a GUI task could be "to create a sparkline for each order number based on data from January to March in the sparkline column," with the corresponding target object being a spreadsheet file. It is understood that this is merely an example, and this disclosure is not limited thereto.

[0085] For example, such as Figure 3 As shown, the operating system task automatic evaluation framework can include a multimodal evaluation criterion evaluation module (as shown in Figure (b)), a sandbox code execution evaluation module (as shown in Figure (c)), and a logic aggregation module (as shown in Figure (d)). The multimodal evaluation criterion evaluation module can be used to perform the trajectory evaluation described above, the sandbox code execution evaluation module can be used to perform the object parsing evaluation described above, and the logic aggregation module can be used to aggregate the two evaluation results.

[0086] like Figure 3As shown in (b) above, the multimodal evaluation criteria module can also be called the multimodal Rubric evaluation module. The multimodal evaluation criteria module can be used to perform multimodal joint evaluation of the execution process and final state of a GUI agent. Exemplary examples include evaluation criteria generation, trajectory correlation screening and visual evidence extraction, rigorous result determination, and multiple rounds of independent evaluation.

[0087] Regarding the steps for generating evaluation criteria, a set of evaluation criteria C={c1,c2,…,c...} can be generated based on the task description. k Furthermore, the set of evaluation criteria can be required to satisfy the mutual independence constraint, that is, the validity of any one evaluation criterion does not directly determine the validity of other criteria. This can be exemplified as P(c i │c j )≈P(c i ) (i≠j), where P(c i ) can represent standard c i The probability that it is true, P(c) i │c j ) can be represented in standard c j Standard c under the condition of being valid i The probability of its validity. This constraint ensures that there are no significant conditional dependencies between the evaluation criteria, thereby guaranteeing that different criteria can independently verify the behavior of the GUI agent and avoiding the cascading misjudgment problem where the failure of a key step leads to the simultaneous failure of multiple evaluation criteria.

[0088] For example, evaluation criteria can cover not only the explicit requirements of the task, but also the implicit constraints and potential requirements of the task under the literal meaning; for example, when the task requires modification of the font style, in addition to verifying whether the font has been modified correctly, it can also verify whether the paragraph layout, alignment and other irrelevant attributes have not been violated.

[0089] In trajectory correlation screening and visual evidence extraction, a "screenshot-standard" correlation matrix can be constructed to model the correlation between multi-timestep screenshots in the execution trajectory and various evaluation standards, and to select the key screenshots most relevant to each standard as visual evidence. For example, a set of interface screenshots O={o1,o2,…,o…} can be constructed. n The correlation matrix R between} and the set of evaluation criteria C, the element in the i-th row and j-th column can be represented as R i , j =Rel(o i ,c j ), where Rel(·) can represent a multimodal relevance scoring function used to measure the semantic relevance between the i-th screenshot and the j-th evaluation criterion.

[0090] For example, for each evaluation criterion c j You can select a predetermined number of key screenshots with the highest relevance to form a key screenshot set. For example, for each criterion... ,choose The most relevant key screenshots are selected to form a key screenshot set. In such an example, visual evidence related to the evaluation criteria can be extracted from each screenshot in the key screenshot set, including information such as changes in interface areas, changes in text attributes, changes in layout states, and interaction results.

[0091] As an example, such a process can be represented as e i , j =Extract(o j , i ,c j ), where e i,j Extract( represents the local visual evidence extracted from the i-th key screenshot.) ) represents the visual evidence extraction function.

[0092] As an example, a strict result-based approach can be adopted, such as using prompts or other methods to prohibit the evaluation agent from using lenient judgment methods like fuzzy matching or semantic approximation matching, and instead requiring the task result to strictly meet the objective constraints. Such an example is particularly suitable for desktop office tasks that typically have clear and deterministic objective requirements, such as, but not limited to, the OSWorld domain.

[0093] Continuing to use desktop office tasks as an example, we can use domain-specific few-shot examples built for office tasks to guide the multimodal evaluation model to learn strict decision logic, thereby improving the model's ability to understand task attributes such as font, layout, and document structure.

[0094] As another non-limiting example, a predetermined number of consecutive screenshots (e.g., five) of the execution trajectory can be used as supplementary visual input to fully observe the interface state changes during the task completion phase. Thus, by using supplementary visual input, the multimodal evaluation model can fully observe the interface state changes during the task completion phase, avoiding misjudgments due to missing key termination steps.

[0095] As an additional, non-limiting embodiment, considering the randomness of the output of the multimodal large language model, a majority voting mechanism can be further introduced. In multiple rounds of independent evaluation, the multimodal evaluation model can be called independently N times (N being an odd number) for the same evaluation task, and a majority vote can be performed on the multiple outputs to obtain the final evaluation result (e.g., trajectory evaluation result S). uvFor example, a trajectory evaluation result can be determined as passed if the number of times a pass is indicated exceeds N / 2 in N evaluations.

[0096] For example, let the result of the t-th independent evaluation be... Therefore, the final output result in trajectory evaluation can be defined as:

[0097] Among them, I( The function is an indicator function that outputs 1 if the number of passes exceeds half of the total number of evaluations, and 0 otherwise.

[0098] Return to reference Figure 3 The following describes an exemplary sandbox code execution evaluation module. As shown in section (c), the sandbox code execution evaluation module can be used to perform structured parsing and comparison of a target object (e.g., a source file) and the target object after the operation (e.g., a processed file) to identify changes in underlying file attributes. For example, a statement-generating agent can autonomously generate a Python script for verifying the file based on task instructions and evaluation criteria generated by the multimodal evaluation module. For example, during the evaluation process, the `run_python` tool can be repeatedly called to execute the code, automatically correcting errors based on standard output (stdout) and standard error (stderr) to achieve autonomous debugging.

[0099] For example, a file semantic difference function can be defined: Δ(F origin ,F final )=Parse(F final ) Parse(F origin ) The `Parse()` function can represent a semantic node extraction function for objects (such as document files, spreadsheet files, and presentation files), used to extract core semantic information such as fonts, paragraphs, table structures, style attributes, and key XML fields. For example, for files like LibreOffice, an XML normalization and semantic node extraction strategy can be further employed, focusing only on core semantic attributes relevant to the task objective and ignoring irrelevant XML rearrangements caused by automatic saving by the software.

[0100] For example, each code execution can run in a child process created in a temporary directory to avoid polluting the host environment.

[0101] For example, a context sliding window mechanism of length w can be used to reduce the computational overhead of long contexts, whose historical context can be represented as H. t ={m t- w+1 ,…,m t-1 ,m t}, where m t This can represent the code execution and feedback information in the t-th round.

[0102] For example, anti-dead-loop mechanisms and forced termination mechanisms can be set up, such as automatically injecting policy constraint prompts when the outputs are consistent multiple times in a row, and forcibly requiring the model to end the evaluation before reaching the maximum number of iterations.

[0103] For example, the sandbox code execution evaluation module can also employ a multi-round independent evaluation and majority voting mechanism to obtain the object resolution evaluation result S. code This is to further improve the robustness and stability of the evaluation system. In such an example, let the result of the t-th independent evaluation be... If ∈{0,1}, then regarding the evaluation of the object resolution result, the final output result can be defined as:

[0104] Return to reference Figure 3 The following describes an exemplary logical merging module, which may also be called a fusion module or logical aggregation module. For example... Figure 3 As shown in section (d), the logic aggregation module can be used to fuse multimodal evaluation results with code execution evaluation results.

[0105] The overall joint evaluation function can be formally represented as: Verifier(T,F origin ,F final )=S UV S CODE In such an example, the overall evaluation result S both It can be represented as S both =S uv ∧S code S uv This can represent the evaluation results of the multimodal evaluation criteria (or the trajectory evaluation results mentioned above), S code This can represent the evaluation result of the sandbox code execution (or the object resolution evaluation result mentioned above). For example, a task can be considered successful only if both results indicate success, thereby reducing the risk of misjudgment due to a single path.

[0106] According to this exemplary embodiment, multimodal evaluation criteria can be performed based on trajectory evaluation, and the operational behavior and visual state changes of the agent can be monitored. According to some additional embodiments, joint evaluation of GUI agent tasks can be further achieved based on differences in the internal structure and attributes of the detected object.

[0107] It is understood that the multimodal evaluation standard assessment module, the sandbox code execution assessment module, and the logic aggregation module can be implemented on the same computing device or distributed across different computing devices; and these modules can be implemented independently or used in combination. It is understood that the above are merely examples, and this disclosure is not limited thereto.

[0108] The following is for reference. Figure 4 This section describes an evaluation method 400 for a GUI operation task according to another exemplary embodiment of the present disclosure. In this embodiment, multimodal evaluation criteria assessment and sandbox code execution assessment can be combined to perform dual-path evaluation on the same GUI operation task and aggregate the results of the two assessments. It is understood that the following description of method 400 is merely a non-limiting example, and embodiments of the present disclosure do not require the simultaneous execution of the dual-path evaluation method, nor do they require the execution of all steps of a single assessment method.

[0109] At step S401, GUI task execution-related data is obtained. For example, the execution-related data may include one or more of execution process data and execution result data. For instance, the task execution trajectory of the task execution agent on the graphical user interface (GUI) operation task, the target object (also called the source object) of the GUI operation task, and the object after the operation (e.g., a processed file) may be obtained. Exemplarily, the task execution trajectory may include a sequence of interface screenshots, and the object after the operation may be obtained by performing at least one operation on the target object through the graphical user interface.

[0110] At step S402, a set of evaluation criteria is determined. For example, the set of evaluation criteria can be determined based on the task description of the GUI operation task. Exemplarily, the set of evaluation criteria may include multiple evaluation criteria that are independent of each other. Exemplarily, the multiple evaluation criteria may include at least one explicit evaluation criterion and at least one implicit evaluation criterion, wherein the at least one explicit evaluation criterion can be used to constrain that at least one target attribute targeted by at least one operation has been successfully changed, and the at least one implicit evaluation criterion can be used to constrain that at least one other attribute besides the target attribute has not been changed. Exemplarily, the set of evaluation criteria may include at least one criterion defining the execution process and at least one criterion defining the execution result.

[0111] In step S403, a set of visual evidence associated with the set of evaluation criteria is obtained. For example, for all evaluation criteria in the set of evaluation criteria, one or more pieces of visual evidence associated with each evaluation criterion can be obtained. For example, visual evidence can be obtained through a sequence of screenshots from the execution process data. For example, a correlation matrix between the sequence of screenshots and the set of evaluation criteria can be constructed, and for each evaluation criterion in the set of evaluation criteria, a first predetermined number of screenshots with the highest correlation are selected to form a set of key screenshots. Then, visual evidence related to each evaluation criterion can be extracted based on each screenshot in the set of key screenshots to obtain a set of visual evidence corresponding to the set of evaluation criteria.

[0112] At step S404, the trajectory evaluation result is obtained. For example, the trajectory evaluation result of the GUI operation task can be determined by a task evaluation agent based on the evaluation criterion set and the visual evidence set, which can be exemplarily represented as S. uv For example, a strict result determination method can be adopted, and the last consecutive second predetermined number of screenshots in the screenshot sequence can be used as supplementary input; for example, N evaluations (N is an odd number greater than 1) can be performed independently for the same task execution trajectory, and the trajectory evaluation result can be determined based on majority voting.

[0113] In step S405, an object parsing statement for evaluating the execution result is obtained. For example, the object parsing statement can be generated by a statement generation agent based on a subset of evaluation criteria related to the execution result from the evaluation criterion set. The object parsing statement can be used to parse the source object and the object after the operation, thereby enabling the extraction of differences.

[0114] In step S406, the object resolution statement is run and debugged in an isolated execution environment until the source resolution result and the processed resolution result are obtained. For example, the method may include running the object resolution statement in a sandbox environment and obtaining execution feedback information, updating the object resolution statement and running it again if the execution feedback information indicates that the object resolution statement needs to be updated, and obtaining the source resolution result for the target object and the processed resolution result for the object after the operation if the object resolution statement runs successfully.

[0115] In step S407, the object parsing evaluation result is obtained. Based on the difference between the source parsing result and the processed parsing result, the object parsing evaluation result of the GUI operation task is determined (which can be exemplarily represented as S). code For example, N evaluations can be performed independently for the same difference, and the evaluation result can be determined based on majority voting.

[0116] At step S408, the comprehensive evaluation result of the GUI operation task is determined. For example, the trajectory evaluation result and the object parsing evaluation result can be aggregated to determine the comprehensive evaluation result of the GUI operation task (exemplarily represented as S). both For example, in response to the determination that both the trajectory evaluation result and the object parsing evaluation result indicate that the task has passed, the overall evaluation result may be determined as passed.

[0117] According to this embodiment, by combining trajectory evaluation based on visual evidence with object parsing evaluation based on object parsing and aggregating the results of the two evaluations, GUI operation tasks can be evaluated from two dimensions: the operation process and the final object. In this example, the two evaluations can be based on a common set of evaluation criteria, making the evaluations of the visual dimension and the object dimension correspond to each other in terms of verification objectives; and the comprehensive evaluation result is determined to be passed only when both evaluations indicate that they have passed, thereby reducing the risk of misjudgment caused by a single path.

[0118] It is understood that the aforementioned multimodal paths (e.g., steps S403 to S404) and code execution paths (e.g., steps S405 to S407) can be executed in parallel or in any order, and this disclosure does not impose any restrictions on this.

[0119] Now for reference Figure 5 This document describes an evaluation apparatus 500 for a GUI operation task according to embodiments of the present disclosure. The evaluation apparatus 500 may include a trajectory acquisition unit 501, a standard acquisition unit 502, an evidence acquisition unit 503, and a determination unit 504. The trajectory acquisition unit 501 is configured to acquire the task execution trajectory of a task execution agent for a graphical user interface (GUI) operation task. The task execution trajectory includes a sequence of screenshots, and the GUI operation task includes performing at least one operation on a target object through the graphical user interface. The standard acquisition unit 502 is configured to determine a set of evaluation criteria based on the task description of the GUI operation task. The set of evaluation criteria includes multiple independent evaluation criteria. The evidence acquisition unit 503 is configured to acquire a set of visual evidence corresponding to the set of evaluation criteria based on the sequence of screenshots. The determination unit 504 is configured to determine the trajectory evaluation result of the GUI operation task by a task evaluation agent based on the set of evaluation criteria and the set of visual evidence.

[0120] The apparatus according to embodiments of the present disclosure can enhance the interpretability of the evaluation process and the accuracy of the evaluation results, thereby helping to achieve more accurate graphical user interface operation.

[0121] The collection, acquisition, storage, use, processing, transmission, provision, and public application of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0122] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0123] refer to Figure 6 The present invention describes a structural block diagram of an electronic device 600 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0124] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0125] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, output unit 607, storage unit 608, and communication unit 609. Input unit 606 can be any type of device capable of inputting information to electronic device 600. Input unit 606 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 607 can be any type of device capable of presenting information, and can include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 608 can include, but is not limited to, disk and optical disk. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0126] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as methods 200 and / or 400 and their variations. For example, in some embodiments, methods 200 and / or 400 and their variations may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of methods 200 and / or 400 and their variations described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute method 200 and / or 400 and its variations by any other suitable means (e.g., by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0129] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0131] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0132] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0133] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0134] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A method for evaluating GUI operation tasks, comprising: Obtain the task execution trajectory of the task execution agent for the graphical user interface (GUI) operation task, wherein the task execution trajectory includes a sequence of interface screenshots, and the GUI operation task includes performing at least one operation on a target object through the graphical user interface; A set of evaluation criteria is determined based on the task description of the GUI operation task, and the set of evaluation criteria includes multiple independent evaluation criteria. Based on the sequence of screenshots, obtain a set of visual evidence corresponding to the set of evaluation criteria; as well as Based on the set of evaluation criteria and the set of visual evidence, the trajectory evaluation result of the GUI operation task is determined by the task evaluation agent.

2. The method according to claim 1, wherein, Obtaining a set of visual evidence corresponding to the set of evaluation criteria includes: Construct a correlation matrix between the sequence of screenshots and the set of evaluation criteria; For each evaluation criterion in the set of evaluation criteria, a first predetermined number of screenshots with the highest relevance are selected to form a set of key screenshots associated with that evaluation criterion; and For each evaluation criterion in the set of evaluation criteria, visual evidence related to that evaluation criterion is extracted based on each screenshot in the set of associated key screenshots.

3. The method according to claim 2, wherein, The visual evidence includes at least one of the following: interface area change features, text attribute change features, layout state change features, and interaction result features.

4. The method according to any one of claims 1-3, wherein, Based on the task description, the set of evaluation criteria includes: Based on the task description, at least one execution requirement for the GUI operation task is determined; and The plurality of evaluation criteria are generated based on the at least one execution requirement, such that the plurality of evaluation criteria satisfy mutual independence constraints, wherein the mutual independence constraints include that the probability of any one of the plurality of evaluation criteria being true is independent of the probability of the other evaluation criteria being true.

5. The method according to claim 4, wherein, The mutual independence constraint includes: the difference between the conditional probability of any evaluation criterion being true under the condition that other evaluation criteria are true and the probability that any evaluation criterion is true is less than a threshold.

6. The method according to any one of claims 1-5, wherein, The set of evaluation criteria includes at least one explicit evaluation criterion and at least one implicit evaluation criterion, wherein the at least one explicit evaluation criterion is used to constrain that the target attribute targeted by the at least one operation of the GUI operation task has been successfully changed, and wherein the at least one implicit evaluation criterion is used to constrain that at least one other attribute besides the target attribute has not been changed.

7. The method according to claim 6, wherein, The target attribute corresponds to a first object feature that is explicitly requested to be modified in the task description, and the at least one other attribute corresponds to at least one second object feature that is not requested to be modified in the task description.

8. The method of claim 7, wherein the target object is a file object, and wherein, The first object feature and the second object feature each contain parameters from at least one of the following dimensions: data content dimension, font and font size dimension, paragraph and spacing dimension, and object and layout dimension.

9. The method according to any one of claims 1-8, wherein, The trajectory evaluation results for the GUI operation task include: Based on the aforementioned set of visual evidence, determine whether the corresponding evaluation criteria are met; and In response to determining that at least one of the plurality of evaluation criteria is not met, the trajectory evaluation result is determined to be unsatisfactory.

10. The method according to any one of claims 1-9, wherein, The trajectory evaluation result of the GUI operation task is also based on the last consecutive second predetermined number of screenshots in the interface screenshot sequence.

11. The method according to any one of claims 1-10, further comprising: The same task evaluation agent independently performs task evaluation N times for the same task execution trajectory to obtain N single evaluation results, where N is an odd number greater than 1; as well as If the number of single evaluation results indicating that the evaluation has passed exceeds N / 2 among the N single evaluation results, the trajectory evaluation result is determined to be passed.

12. The method according to any one of claims 1-11, further comprising: Obtain object parsing evaluation results, which are obtained by parsing and comparing the unprocessed target object before performing the at least one operation with the processed target object after performing the at least one operation; as well as In response to the determination that both the trajectory evaluation result and the object parsing evaluation result indicate that the task has passed, the comprehensive evaluation result is determined to be passed.

13. The method according to claim 12, in, The object resolution evaluation results include: Obtain at least one object resolution evaluation criterion; Based on the object parsing evaluation criteria, an object parsing statement is obtained, which is used to parse the unprocessed target object and the processed target object; Run the object resolution statement in an isolated execution environment to obtain the source resolution result for the unprocessed target object and the post-processing resolution result for the processed target object; and The object parsing evaluation result is determined based on the difference between the source parsing result and the processed parsing result.

14. The method according to claim 13, wherein, Obtaining at least one object resolution evaluation criterion includes obtaining a subset of the evaluation criteria set that is relevant to the execution result.

15. An evaluation device for GUI operation tasks, comprising: The trajectory acquisition unit is used to acquire the task execution trajectory of the task execution agent on the graphical user interface (GUI) operation task. The task execution trajectory includes a sequence of interface screenshots. The GUI operation task includes performing at least one operation on a target object through the graphical user interface. The standard acquisition unit is used to determine a set of evaluation standards based on the task description of the GUI operation task, wherein the set of evaluation standards includes multiple independent evaluation standards. An evidence acquisition unit is used to obtain a set of visual evidence corresponding to the set of evaluation criteria based on the sequence of screenshots of the interface. The determining unit is used to determine the trajectory evaluation result of the GUI operation task based on the evaluation standard set and the visual evidence set, through a task evaluation agent.

16. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-14.

17. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-14.

18. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-14.

Images