Multi-dimensional graphical proxy interface integration evaluation method and system

Abstract

The invention provides a multi-dimensional graphical agent interface integration evaluation method and system, and the method comprises the steps: carrying out the graphical agent evaluation interface integration of a multi-dimensional system through the standardized process and input and output representation of a cross-operating system and an application program; structured input and output are defined through linear sequential logic LTL grammar, multi-dimensional decomposition is carried out on the agent core capability of a graphical user interface (GUI), and a multi-dimensional decomposition quantitative evaluation model is established; on the basis of the multi-dimensional decomposition quantitative evaluation model, fine-grained indexes based on linear sequential logic LTL grammar analysis are set, and a multi-dimensional automatic evaluation system from bottom-layer action sequence matching to high-layer task success rate is constructed; and according to the multi-dimensional automatic evaluation system, constructing an integrated graphic user interface Unify GUI evaluation data set comprising a plurality of cross-platform tasks, and carrying out multi-dimensional evaluation on the plurality of cross-platform tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of computer system cross-program interface integrated intelligent multi-dimensional quantitative evaluation testing technology, and more specifically, the present invention relates to a multi-dimensional graphical agent interface integrated evaluation method and system. Background Art

[0002] At present, the evaluation of graphical agent interfaces faces the following problems: dimensional fragmentation, including the separation of evaluation of planning, positioning and environmental adaptability; input and output heterogeneity, including the mixing of natural language, multimodal input and code, options and predefined action output; non-standardization of evaluation indicators, including the mixing of action sequence matching and task success rate; fragmentation of capability evaluation dimensions, including the existing benchmarks focusing on a single capability, such as VideoGUI only evaluating planning capabilities; lack of cross-platform comparability, such as different applications on Windows, Android and Linux using different input representations, such as screenshots and accessible trees, and the lack of detailed granularity of output actions; insufficient interpretability of indicators, over-reliance on large language models to generate scores, and other issues remain to be resolved; therefore, it is necessary to propose a multi-dimensional graphical agent interface integrated evaluation method and system to at least partially solve the problems existing in the existing technology. Summary of the Invention

[0003] A series of simplified concepts are introduced in the summary of the invention, which will be further explained in detail in the specific implementation method. The summary of the invention does not mean to attempt to limit the key features and necessary technical features of the technical solution for protection, nor does it mean to attempt to determine the scope of protection of the technical solution for protection.

[0004] To at least partially address the above-mentioned problems, the present invention provides a multi-dimensional graphical agent interface integrated evaluation method, comprising: S10, graphical agent evaluation interface integration for multi-dimensional systems through standardized processes and input-output representations across operating systems and applications; S20, defines structured input and output through linear temporal logic (LTL) syntax, decomposes the core capabilities of the graphical user interface (GUI) agent into multiple dimensions, and establishes a multi-dimensional decomposition quantitative evaluation model; S30, based on a multi-dimensional decomposition quantitative evaluation model, sets fine-grained indicators based on linear temporal logic (LTL) syntax parsing, and builds a multi-dimensional automated evaluation system from low-level action sequence matching to high-level task success rate; S40, based on the multi-dimensional automated evaluation system, builds an integrated graphical user interface Unify GUI evaluation dataset containing multiple cross-platform tasks, and conducts multi-dimensional evaluation of multiple cross-platform tasks.

[0005] Preferably, S10 includes: S101, which uses linear temporal logic to implement standardized processes and input-output representations across operating systems and applications, enabling natural language instruction task input and structured output; S102, integrating a graphical agent evaluation interface of a multi-dimensional system; the integration of a graphical agent evaluation interface of a multi-dimensional system includes: natural language, JSON and Python code integration.

[0006] Preferably, S20 includes: S201, defining structured input and output using linear temporal logic (LTL) syntax, multi-dimensionally decomposing the core capabilities of the graphical user interface (GUI) agent, and obtaining a graphical user interface (GUI) multi-dimensional decomposition module group; S202, establishing a multidimensional decomposition quantitative evaluation model according to the graphical user interface GUI multidimensional decomposition module group; The graphical user interface (GUI) multidimensional decomposition module group includes: a task understanding module, a planning generation module, an action positioning module, and a state estimation module; these modules respectively analyze and interpret task instructions in natural language format, expand the target state description into multiple intermediate states, convert the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predict the system state after executing the action sequence based on the observation of the current environment and the action sequence.

[0007] Preferably, S30 includes: S301: Based on the multi-dimensional decomposition quantitative evaluation model, set fine-grained system indicators based on linear temporal logic (LTL) syntax parsing; S302: Based on the fine-grained system indicators, the interface input and output abstract definition layer is run to conduct multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, thus building a multi-dimensional automated evaluation system. Fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators and operational feasibility granularity indicators.

[0008] Preferably, S40 includes: S401, according to the LTL grammar rules, set a grammar parser to convert the generated text sequence that complies with the LTL grammar rules into an abstract syntax tree; S402: Based on the abstract syntax tree, the LTL formula generated by the agent and visual language large model is parsed to evaluate the structure and nodes, build an evaluation execution layer, and construct an integrated graphical user interface Unify GUI evaluation dataset that includes multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks.

[0009] The present invention provides a multi-dimensional graphical agent interface integrated evaluation system, comprising: Multi-dimensional system interface integration subsystem, which integrates graphical agent evaluation interfaces of multi-dimensional systems through standardized processes and input and output representations across operating systems and applications; The multi-dimensional decomposition quantitative evaluation subsystem defines structured input and output using linear temporal logic (LTL) syntax, decomposes the core capabilities of the graphical user interface (GUI) agent into multiple dimensions, and establishes a multi-dimensional decomposition quantitative evaluation model. The sequence matching evaluation system subsystem is based on a multi-dimensional decomposition quantitative evaluation model, sets fine-grained indicators based on linear temporal logic (LTL) syntax parsing, and builds a multi-dimensional automated evaluation system from low-level action sequence matching to high-level task success rate; The cross-platform evaluation dataset subsystem builds the Unify GUI evaluation dataset, which includes multiple cross-platform tasks, based on a multi-dimensional automated evaluation system, and conducts multi-dimensional evaluations of multiple cross-platform tasks.

[0010] Preferably, the multi-dimensional system interface integration subsystem includes: The linear temporal language instruction subsystem uses linear temporal logic to implement standardized processes and input and output representations across operating systems and applications, enabling natural language instruction task input and structured output; The multi-dimensional cross-platform interface integration subsystem integrates the graphical agent evaluation interface of the multi-dimensional system; the graphical agent evaluation interface integration of the multi-dimensional system includes: natural language, JSON and Python code integration.

[0011] Preferably, the multi-dimensional decomposition quantitative evaluation subsystem includes: The interface agent multi-dimensional decomposition subsystem defines structured input and output through linear temporal logic (LTL) syntax, multi-dimensionally decomposes the core capabilities of the graphical user interface (GUI) agent, and obtains a graphical user interface (GUI) multi-dimensional decomposition module group; The decomposition and quantitative evaluation model subsystem establishes a multi-dimensional decomposition and quantitative evaluation model based on the multi-dimensional decomposition module group of the graphical user interface GUI; The graphical user interface (GUI) multidimensional decomposition module group includes: a task understanding module, a planning generation module, an action positioning module, and a state estimation module; these modules respectively analyze and interpret task instructions in natural language format, expand the target state description into multiple intermediate states, convert the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predict the system state after executing the action sequence based on the observation of the current environment and the action sequence.

[0012] Preferably, the sequence matching assessment system subsystem includes: The fine-grained system indicator subsystem sets fine-grained system indicators based on linear temporal logic (LTL) syntax parsing based on a multi-dimensional decomposition quantitative evaluation model. The interface operation multi-dimensional evaluation subsystem runs the interface input and output abstract definition layer based on fine-grained system indicators, conducts multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, and builds a multi-dimensional automated evaluation system; Fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators and operational feasibility granularity indicators.

[0013] Preferably, the cross-platform evaluation dataset subsystem includes: The abstract syntax tree sequence conversion subsystem sets a syntax parser according to the LTL syntax rules and converts the generated text sequence that conforms to the LTL syntax rules into an abstract syntax tree; The syntax parsing cross-platform evaluation subsystem parses the LTL formulas generated by the agent and visual language large model based on the abstract syntax tree, evaluates the structure and nodes, builds the evaluation execution layer, and constructs the Unify GUI evaluation dataset that includes multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks.

[0014] Compared with the prior art, the present invention has at least the following beneficial effects: The present invention provides a multi-dimensional graphical agent interface integration evaluation method and system. The method integrates the graphical agent evaluation interface of a multi-dimensional system through standardized processes and input and output representations across operating systems and applications. The method defines structured input and output using linear temporal logic (LTL) syntax, multi-dimensionally decomposes the core capabilities of the graphical user interface (GUI) agent, and establishes a multi-dimensional decomposition quantitative evaluation model. Based on the multi-dimensional decomposition quantitative evaluation model, fine-grained indicators based on linear temporal logic (LTL) syntax parsing are set to construct a multi-dimensional automated evaluation system from bottom-level action sequence matching to high-level task success rate. Based on the multi-dimensional automated evaluation system, an integrated graphical user interface (Unify) containing multiple cross-platform tasks is constructed. A GUI evaluation dataset is used to conduct multi-dimensional evaluations of multiple cross-platform tasks. A systematic evaluation of existing large visual language models and graphical agents reveals key flaws in current GUI agents, such as hallucination errors, positioning errors, and system modeling errors, in the dimensions of task understanding, plan generation, action localization, and system modeling. The evaluation interface for graphical agents in multi-dimensional systems is integrated: standardized processes and input-output representations across operating systems and applications are implemented using first-order linear temporal logic (LTL), enabling the integration of natural language instruction task input and structured output (natural language, JSON, or Python code). The four major capability modules of graphical agents are formally clarified: the core capabilities of GUI agents are decomposed into a task understanding module, a plan generation module, an action localization module, and a state estimation module, and a model is established. Each module defines structured input and output using LTL syntax, constructing interpretable quantitative evaluation indicators. A systematic evaluation system is developed: fine-grained indicators based on LTL syntax parsing are designed to significantly improve the accuracy and comprehensiveness of multi-dimensional automated evaluations, from low-level action sequence matching to high-level task success rates. This invention has important technical significance and significant effects.

[0015] The multi-dimensional graphical agent interface integrated evaluation method and system described in the present invention, other advantages, objectives and features of the present invention will be reflected in part through the following description, and in part will be understood by technical personnel in this field through research and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a diagram illustrating an embodiment of a multi-dimensional graphical agent interface integrated evaluation system architecture according to the present invention.

[0017] Figure 2 This is a diagram illustrating an embodiment of a multi-dimensional graphical agent interface integrated evaluation system status according to the present invention.

[0018] Figure 3 This is a diagram illustrating an embodiment of a multi-dimensional graphical agent interface integrated evaluation method and system expanded syntax representation type according to the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments so that those skilled in the art can implement the invention with reference to the description. Figure 1-Figure 3 As shown, the present invention provides a multi-dimensional graphical agent interface integrated evaluation method, comprising: S10, graphical agent evaluation interface integration for multi-dimensional systems through standardized processes and input-output representations across operating systems and applications; S20, defines structured input and output through linear temporal logic (LTL) syntax, decomposes the core capabilities of the graphical user interface (GUI) agent into multiple dimensions, and establishes a multi-dimensional decomposition quantitative evaluation model; S30, based on a multi-dimensional decomposition quantitative evaluation model, sets fine-grained indicators based on linear temporal logic (LTL) syntax parsing, and builds a multi-dimensional automated evaluation system from low-level action sequence matching to high-level task success rate; S40, based on the multi-dimensional automated evaluation system, builds an integrated graphical user interface Unify GUI evaluation dataset containing multiple cross-platform tasks, and conducts multi-dimensional evaluation of multiple cross-platform tasks.

[0020] The principle and effect of the above technical solution are as follows: the present invention provides a multi-dimensional graphical agent interface integration evaluation method, including: integrating the graphical agent evaluation interface of the multi-dimensional system through standardized processes and input and output representations across operating systems and applications; defining structured input and output through linear temporal logic LTL syntax, multi-dimensionally decomposing the core capabilities of the graphical user interface GUI agent, and establishing a multi-dimensional decomposition quantitative evaluation model; based on the multi-dimensional decomposition quantitative evaluation model, setting fine-grained indicators based on linear temporal logic LTL syntax parsing, and constructing a multi-dimensional automated evaluation system from bottom-level action sequence matching to high-level task success rate; based on the multi-dimensional automated evaluation system, constructing an integrated graphical user interface Unify that includes multiple cross-platform tasks GUI evaluation dataset, conducts multi-dimensional evaluation of multiple cross-platform tasks; through systematic evaluation of existing visual language large models and graphical agents, reveals the key defects of current GUI agents in task understanding, plan generation, action positioning and system modeling, such as illusion errors, positioning errors and system modeling errors; Graphical agent evaluation interface integration of multi-dimensional systems: through first-order linear temporal logic (LTL) to achieve standardized processes and input and output representation across operating systems and applications, and realize the integration of natural language instruction task input and structured output including natural language, JSON and Python code; formally clarify the four major capability modules of graphical agents: decompose the core capabilities of GUI agents into task understanding module, plan generation module, action positioning module and state estimation module Blocks are built, and each module defines structured input and output through LTL syntax, and constructs interpretable quantitative evaluation indicators; a systematic evaluation system: fine-grained indicators based on LTL syntax parsing are designed to achieve multi-dimensional automated evaluation accuracy and comprehensiveness from low-level action sequence matching to high-level task success rate; fine-grained indicators include grammatical correctness, logical consistency and operational feasibility; the Unify-GUI of the present invention includes: an operating system and a test evaluation dataset corresponding to multiple applications and multiple specific graphical tasks; each application is pre-set with a series of complete state prepositions and action prepositions to describe the goals and plans of different tasks; in the Unify-GUI dataset, the complete task goals and task plans are marked with the ground truth (ground truth) for each test task truth), and conduct formal evaluation. In addition, by executing tasks, recording operation processes and interface screenshots, multiple pairs of test data are screened and constructed to conduct action positioning tests and status tests. The operating system includes Linux; multiple applications include PowerPoint, Excel, Word, OS, and Chrome. The multiple pairs of test data include 900 pairs of test data. The Unify-GUI of the present invention conducts a multi-dimensional graphical agent interface integration evaluation, evaluating the four-dimensional capabilities of the graphical agent in completing graphical interface tasks. The present invention has important technical significance and significant effects.

[0021] In one embodiment, S10 includes: S101, which uses linear temporal logic to implement standardized processes and input-output representations across operating systems and applications, enabling natural language instruction task input and structured output; S102, integrating a graphical agent evaluation interface of a multi-dimensional system; the integration of a graphical agent evaluation interface of a multi-dimensional system includes: natural language, JSON and Python code integration.

[0022] The principles and effects of the above technical solution are: to achieve standardized processes and input-output representations across operating systems and applications through linear temporal logic, and to achieve natural language instruction task input and structured output; to integrate the graphical agent evaluation interface of multi-dimensional systems; the graphical agent evaluation interface integration of multi-dimensional systems includes: natural language, JSON and Python code integration; the graphical agent evaluation interface integration of multi-dimensional systems: to achieve standardized processes and input-output representations across operating systems and applications through first-order linear temporal logic (LTL), and to achieve natural language instruction task input and structured output including natural language, JSON and Python code integration.

[0023] In one embodiment, S20 includes: S201, defining structured input and output using linear temporal logic (LTL) syntax, multi-dimensionally decomposing the core capabilities of the graphical user interface (GUI) agent, and obtaining a graphical user interface (GUI) multi-dimensional decomposition module group; S202, establishing a multidimensional decomposition quantitative evaluation model according to the graphical user interface GUI multidimensional decomposition module group; The graphical user interface (GUI) multidimensional decomposition module group includes: a task understanding module, a planning generation module, an action positioning module, and a state estimation module; these modules respectively analyze and interpret task instructions in natural language format, expand the target state description into multiple intermediate states, convert the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predict the system state after executing the action sequence based on the observation of the current environment and the action sequence.

[0024] The principle and effect of the above technical solution are as follows: by defining structured input and output using linear temporal logic (LTL) syntax, the core capabilities of the graphical user interface (GUI) agent are multidimensionally decomposed to obtain a GUI multidimensional decomposition module group; based on the GUI multidimensional decomposition module group, a multidimensional decomposition quantitative evaluation model is established; The graphical user interface (GUI) multi-dimensional decomposition module group includes: a task understanding module, a plan generation module, an action positioning module, and a state estimation module; each module analyzes and interprets task instructions in natural language format, expands the target state description into multiple intermediate states, converts the intermediate states into a series of strings that realize the intermediate states to form an action sequence, and predicts the system state after executing the action sequence based on observations of the current environment and the action sequence; Task understanding module: Analyzes and interprets task instructions in natural language format, converts operation commands into formalized strings that conform to LTL syntax structure, describes the target state, and obtains the target state description; Planning generation module: This module expands the target state description into multiple intermediate states. The intermediate states include the necessary processes to achieve the final goal and conform to the LTL syntax structure. Action positioning module: converts the intermediate state into a series of strings that realize the intermediate state to form an action sequence; State Estimation Module: predicts the system state after executing an action sequence based on observations of the current environment and the action sequence; evaluates the execution of the current task and the agent's understanding of the system state.

[0025] In one embodiment, S30 includes: S301: Based on the multi-dimensional decomposition quantitative evaluation model, set fine-grained system indicators based on linear temporal logic (LTL) syntax parsing; S302: Based on the fine-grained system indicators, the interface input and output abstract definition layer is run to conduct multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, thus building a multi-dimensional automated evaluation system. Fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators and operational feasibility granularity indicators.

[0026] The principle and effect of the above technical solution are as follows: Based on a multi-dimensional decomposition quantitative evaluation model, fine-grained system indicators based on linear temporal logic (LTL) syntax parsing are set; based on the fine-grained system indicators, the interface input and output abstract definition layer is run, and multi-dimensional automated evaluation is performed from low-level action sequence matching to high-level task success rate, thus building a multi-dimensional automated evaluation system; the fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators, and operational feasibility granularity indicators; the interface input and output abstract definition layer operation includes: Task understanding module: Input includes instructions in natural language format and observations of the current environment; output includes a target state description string in LTL syntax format; Planning generation module: The input includes a target state description string in LTL syntax format and observations of the current environment; the output includes an intermediate process state description string in LTL syntax format; Action positioning module: The input includes a string describing the intermediate process state in LTL syntax and observations of the current environment; the output includes a sequence of atomic actions; State estimation module: The input includes the atomic action sequence and the observation of the current environment; the output includes the expected system state after the action sequence is executed; Action types include: click, double click, hot-key, scroll, drag, or right click.

[0027] In one embodiment, S40 includes: S401, according to the LTL grammar rules, set a grammar parser to convert the generated text sequence that complies with the LTL grammar rules into an abstract syntax tree; S402: Based on the abstract syntax tree, the LTL formula generated by the agent and visual language large model is parsed to evaluate the structure and nodes, build an evaluation execution layer, and construct an integrated graphical user interface Unify GUI evaluation dataset that includes multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks.

[0028] The principle and effect of the above technical solution are as follows: according to the LTL grammar rules, a grammar parser is set to convert the generated text sequence that conforms to the LTL grammar rules into an abstract syntax tree; according to the abstract syntax tree, the LTL formula generated by the agent and the visual language large model is parsed to achieve the evaluation of the structure and nodes, and the evaluation execution layer is built to construct the Unify GUI evaluation data set of the integrated graphical user interface containing multiple cross-platform tasks, and multiple cross-platform tasks are evaluated in multiple dimensions; such as Figure 3As shown; according to the LTL grammar rules, a grammar parser is set to convert the generated text sequence that conforms to the LTL grammar rules into an abstract syntax tree, including: setting LTL grammar generation rules to formalize the input and output into an LTL grammar sequence (sequence), which contains state conditions and action conditions connected by the connecting symbol THEN, and the two are respectively composed of predefined state prepositions (state proposition) and action prepositions (action proposition); these two types of prepositions require different parameters according to specific circumstances; the grammar representation type is expanded through the value function in the parameter; value function; grammar parsing method: using Python program to perform lexical analysis and grammatical analysis, converting the generated text sequence that conforms to the LTL grammar rules into an abstract syntax tree; the abstract syntax tree includes a hierarchical JSON structure document; implementing the evaluation of structure and nodes; The types of parameters expressed by expanding the syntax through value functions include: when the true value of the parameter is affected by the evaluation time, such as setting next week or the next Tuesday in the instance task, and the specific true value of the parameter cannot be determined before the test; when the true value of the parameter is affected by the result of a specific numerical calculation, such as setting the sum of a column or the average of a row in the task, the numerical result depends on the intermediate state result during the operation, or it cannot be accurately calculated in the current task understanding stage, and the value function setting includes sum representing the sum of a column or average representing the average of a row; the value function setting also includes relative time one, the first relative time value model representing next week, or relative time two, the second relative time value model representing the next Tuesday; The indicator calculation includes: grammatical correctness, logical consistency, operational feasibility, action positioning accuracy and system state prediction accuracy; the Unify GUI evaluation dataset of integrated graphical user interface for multiple cross-platform tasks contains the Unify GUI evaluation dataset of integrated graphical user interface for 233 cross-platform tasks.

[0029] The present invention provides a multi-dimensional graphical agent interface integrated evaluation system, comprising: Multi-dimensional system interface integration subsystem, which integrates graphical agent evaluation interfaces of multi-dimensional systems through standardized processes and input and output representations across operating systems and applications; The multi-dimensional decomposition quantitative evaluation subsystem defines structured input and output using linear temporal logic (LTL) syntax, decomposes the core capabilities of the graphical user interface (GUI) agent into multiple dimensions, and establishes a multi-dimensional decomposition quantitative evaluation model. The sequence matching evaluation system subsystem is based on a multi-dimensional decomposition quantitative evaluation model, sets fine-grained indicators based on linear temporal logic (LTL) syntax parsing, and builds a multi-dimensional automated evaluation system from low-level action sequence matching to high-level task success rate; The cross-platform evaluation dataset subsystem builds the Unify GUI evaluation dataset, which includes multiple cross-platform tasks, based on a multi-dimensional automated evaluation system, and conducts multi-dimensional evaluations of multiple cross-platform tasks.

[0030] The principle and effect of the above technical solution are as follows: the present invention provides a multi-dimensional graphical agent interface integration evaluation system, including: a multi-dimensional system interface integration subsystem, which integrates the graphical agent evaluation interface of the multi-dimensional system through standardized processes and input and output representations across operating systems and applications; a multi-dimensional decomposition quantitative evaluation subsystem, which defines structured input and output through linear temporal logic LTL syntax, multi-dimensionally decomposes the core capabilities of the graphical user interface GUI agent, and establishes a multi-dimensional decomposition quantitative evaluation model; a sequence matching evaluation system subsystem, which sets fine-grained indicators based on linear temporal logic LTL syntax parsing based on the multi-dimensional decomposition quantitative evaluation model, and constructs a multi-dimensional automated evaluation system from bottom-level action sequence matching to high-level task success rate; a cross-platform evaluation data set subsystem, which constructs an integrated graphical user interface Unify containing multiple cross-platform tasks based on the multi-dimensional automated evaluation system. GUI evaluation dataset, conducts multi-dimensional evaluation of multiple cross-platform tasks; through systematic evaluation of existing visual language large models and graphical agents, reveals the key defects of current GUI agents in task understanding, plan generation, action positioning and system modeling, such as illusion errors, positioning errors and system modeling errors; Graphical agent evaluation interface integration of multi-dimensional systems: through first-order linear temporal logic (LTL) to achieve standardized processes and input and output representation across operating systems and applications, and realize the integration of natural language instruction task input and structured output including natural language, JSON and Python code; formally clarify the four major capability modules of graphical agents: decompose the core capabilities of GUI agents into task understanding module, plan generation module, action positioning module and state estimation module Block, build a model, each module defines structured input and output through LTL syntax, and constructs interpretable quantitative evaluation indicators; systematic evaluation system: design fine-grained indicators based on LTL syntax parsing, and realize multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, with significantly improved accuracy and comprehensiveness; fine-grained indicators include grammatical correctness, logical consistency and operational feasibility; the Unify-GUI of the present invention includes: a test evaluation data set for an operating system and multiple specific graphical tasks corresponding to multiple applications; each application pre-sets a series of complete state prepositions and action prepositions to describe the goals and plans of different tasks; in the Unify-GUI data set, the complete task goals and task plans are marked with the ground truth for each test task for formal evaluation; in addition, by executing the task, recording the operation process and interface screenshots, multiple pairs of test data are screened and constructed to perform action positioning tests and state tests; the operating system includes Linux; multiple applications include PowerPoint, Excel, Word, OS and Chrome; the multiple pairs of test data include 900 pairs of test data;The Unify-GUI of the present invention conducts a multi-dimensional graphical agent interface integration assessment, evaluating the four-dimensional capabilities of the graphical agent in completing graphical interface tasks. The present invention has important technical significance and significant effects.

[0031] In one embodiment, the multi-dimensional system interface integration subsystem includes: The linear temporal language instruction subsystem uses linear temporal logic to implement standardized processes and input and output representations across operating systems and applications, enabling natural language instruction task input and structured output; The multi-dimensional cross-platform interface integration subsystem integrates the graphical agent evaluation interface of the multi-dimensional system; the graphical agent evaluation interface integration of the multi-dimensional system includes: natural language, JSON and Python code integration.

[0032] The principles and effects of the above technical solution are: a multi-dimensional system interface integration subsystem, including: a linear temporal language instruction subsystem, which implements standardized processes and input-output representations across operating systems and applications through linear temporal logic, and realizes natural language instruction task input and structured output; a multi-dimensional cross-platform interface integration subsystem, which performs graphical agent evaluation interface integration of multi-dimensional systems; the graphical agent evaluation interface integration of multi-dimensional systems includes: natural language, JSON and Python code integration; the graphical agent evaluation interface integration of multi-dimensional systems: through first-order linear temporal logic (LTL) to achieve standardized processes and input-output representations across operating systems and applications, and realize natural language instruction task input and structured output including natural language, JSON and Python code integration.

[0033] In one embodiment, the multi-dimensional decomposition quantitative evaluation subsystem includes: The interface agent multi-dimensional decomposition subsystem defines structured input and output through linear temporal logic (LTL) syntax, multi-dimensionally decomposes the core capabilities of the graphical user interface (GUI) agent, and obtains a graphical user interface (GUI) multi-dimensional decomposition module group; The decomposition and quantitative evaluation model subsystem establishes a multi-dimensional decomposition and quantitative evaluation model based on the multi-dimensional decomposition module group of the graphical user interface GUI; The graphical user interface (GUI) multidimensional decomposition module group includes: a task understanding module, a planning generation module, an action positioning module, and a state estimation module; these modules respectively analyze and interpret task instructions in natural language format, expand the target state description into multiple intermediate states, convert the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predict the system state after executing the action sequence based on the observation of the current environment and the action sequence.

[0034] The principle and effect of the above technical solution are as follows: the multi-dimensional decomposition quantitative evaluation subsystem includes: the interface agent multi-dimensional decomposition subsystem, which defines structured input and output through linear temporal logic (LTL) syntax, multi-dimensionally decomposes the core capabilities of the graphical user interface (GUI) agent, and obtains the graphical user interface (GUI) multi-dimensional decomposition module group; The decomposition and quantitative evaluation model subsystem establishes a multidimensional decomposition and quantitative evaluation model based on the graphical user interface (GUI) multidimensional decomposition module group; the GUI multidimensional decomposition module group includes: a task understanding module, a plan generation module, an action positioning module, and a state estimation module; each module analyzes and interprets task instructions in natural language format, expands the target state description into multiple intermediate states, converts the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predicts the system state after executing the action sequence based on observations of the current environment and the action sequence; Task understanding module: Analyzes and interprets task instructions in natural language format, converts operation commands into formalized strings that conform to LTL syntax structure, describes the target state, and obtains the target state description; Planning generation module: This module expands the target state description into multiple intermediate states. The intermediate states include the necessary processes to achieve the final goal and conform to the LTL syntax structure. Action positioning module: converts the intermediate state into a series of strings that realize the intermediate state to form an action sequence; State Estimation Module: predicts the system state after executing an action sequence based on observations of the current environment and the action sequence; evaluates the execution of the current task and the agent's understanding of the system state.

[0035] In one embodiment, the sequence matching assessment subsystem comprises: The fine-grained system indicator subsystem sets fine-grained system indicators based on linear temporal logic (LTL) syntax parsing based on a multi-dimensional decomposition quantitative evaluation model. The interface operation multi-dimensional evaluation subsystem runs the interface input and output abstract definition layer based on fine-grained system indicators, conducts multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, and builds a multi-dimensional automated evaluation system; Fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators and operational feasibility granularity indicators.

[0036] The principles and effects of the above technical solution are as follows: the sequence matching evaluation system subsystem includes: a fine-grained system indicator subsystem, which sets fine-grained system indicators based on linear temporal logic (LTL) syntax parsing based on a multi-dimensional decomposition quantitative evaluation model; an interface operation multi-dimensional evaluation subsystem, which runs the interface input and output abstract definition layer based on the fine-grained system indicators, and conducts multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, thus building a multi-dimensional automated evaluation system; Fine-grained system indicators include: syntax correctness granularity indicators, logical consistency granularity indicators, and operational feasibility granularity indicators; interface input and output abstract definition layer operations include: Task understanding module: Input includes instructions in natural language format and observations of the current environment; output includes a target state description string in LTL syntax format; Planning generation module: The input includes a target state description string in LTL syntax format and observations of the current environment; the output includes an intermediate process state description string in LTL syntax format; Action positioning module: The input includes a string describing the intermediate process state in LTL syntax and observations of the current environment; the output includes a sequence of atomic actions; State estimation module: The input includes the atomic action sequence and the observation of the current environment; the output includes the expected system state after the action sequence is executed; Action types include: click, double click, hot-key, scroll, drag, or right click.

[0037] In one embodiment, the cross-platform evaluation dataset subsystem includes: The abstract syntax tree sequence conversion subsystem sets a syntax parser according to the LTL syntax rules and converts the generated text sequence that conforms to the LTL syntax rules into an abstract syntax tree; The syntax parsing cross-platform evaluation subsystem parses the LTL formulas generated by the agent and visual language large model based on the abstract syntax tree, evaluates the structure and nodes, builds the evaluation execution layer, and constructs the Unify GUI evaluation dataset that includes multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks.

[0038] The principle and effect of the above technical solution are as follows: the cross-platform evaluation dataset subsystem includes: the abstract syntax tree sequence conversion subsystem, which sets a syntax parser according to the LTL syntax rules to convert the generated text sequence that conforms to the LTL syntax rules into an abstract syntax tree; the syntax parsing cross-platform evaluation subsystem, which performs syntax tree parsing on the LTL formulas generated by the agent and visual language large model according to the abstract syntax tree, realizes the evaluation of the structure and nodes, builds the evaluation execution layer, and constructs the Unify GUI evaluation dataset containing multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks; such as Figure 3 As shown; according to the LTL grammar rules, a grammar parser is set to convert the generated text sequence that conforms to the LTL grammar rules into an abstract syntax tree, including: setting LTL grammar generation rules to formalize the input and output into an LTL grammar sequence (sequence), which contains state conditions and action conditions connected by the connecting symbol THEN, and the two are respectively composed of predefined state prepositions (state proposition) and action prepositions (action proposition); these two types of prepositions require different parameters according to specific circumstances; the grammar representation type is expanded through the value function in the parameter; value function; grammar parsing method: using Python program to perform lexical analysis and grammatical analysis, converting the generated text sequence that conforms to the LTL grammar rules into an abstract syntax tree; the abstract syntax tree includes a hierarchical JSON structure document; implementing the evaluation of structure and nodes; The types of parameters expressed by expanding the syntax through value functions include: when the true value of the parameter is affected by the evaluation time, such as setting next week or the next Tuesday in the instance task, and the specific true value of the parameter cannot be determined before the test; when the true value of the parameter is affected by the result of a specific numerical calculation, such as setting the sum of a column or the average of a row in the task, the numerical result depends on the intermediate state result during the operation, or it cannot be accurately calculated in the current task understanding stage, and the value function setting includes sum representing the sum of a column or average representing the average of a row; the value function setting also includes relative time one, the first relative time value model representing next week, or relative time two, the second relative time value model representing the next Tuesday; The indicator calculation includes: grammatical correctness, logical consistency, operational feasibility, action positioning accuracy and system state prediction accuracy; the Unify GUI evaluation dataset of integrated graphical user interface for multiple cross-platform tasks contains the Unify GUI evaluation dataset of integrated graphical user interface for 233 cross-platform tasks.

[0039] Although the embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the description and implementation methods. They can be fully applied to various fields suitable for the present invention. For those familiar with the art, additional modifications can be easily implemented. Therefore, without departing from the general concept defined by the claims and the scope of equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A multi-dimensional graphical agent interface integration evaluation method, characterized in that: include: S10, graphical agent evaluation interface integration for multi-dimensional systems through standardized processes and input-output representations across operating systems and applications; S20, defines structured input and output through linear temporal logic (LTL) syntax, decomposes the core capabilities of the graphical user interface (GUI) agent into multiple dimensions, and establishes a multi-dimensional decomposition quantitative evaluation model; S30, based on a multi-dimensional decomposition quantitative evaluation model, sets fine-grained indicators based on linear temporal logic (LTL) syntax parsing, and builds a multi-dimensional automated evaluation system from low-level action sequence matching to high-level task success rate; S40, based on the multi-dimensional automated evaluation system, builds an integrated graphical user interface Unify GUI evaluation dataset containing multiple cross-platform tasks, and conducts multi-dimensional evaluation of multiple cross-platform tasks.

2. A multi-dimensional graphical agent interface integration evaluation method according to claim 1, characterized in that: The S10 includes: S101, which uses linear temporal logic to implement standardized processes and input-output representations across operating systems and applications, enabling natural language instruction task input and structured output; S102, integrating a graphical agent evaluation interface of a multi-dimensional system; the integration of a graphical agent evaluation interface of a multi-dimensional system includes: natural language, JSON and Python code integration.

3. A multi-dimensional graphical agent interface integration evaluation method according to claim 1, characterized in that: The S20 includes: S201, defining structured input and output using linear temporal logic (LTL) syntax, multi-dimensionally decomposing the core capabilities of the graphical user interface (GUI) agent, and obtaining a graphical user interface (GUI) multi-dimensional decomposition module group; S202, establishing a multidimensional decomposition quantitative evaluation model according to the graphical user interface GUI multidimensional decomposition module group; The graphical user interface (GUI) multidimensional decomposition module group includes: a task understanding module, a planning generation module, an action positioning module, and a state estimation module; these modules respectively analyze and interpret task instructions in natural language format, expand the target state description into multiple intermediate states, convert the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predict the system state after executing the action sequence based on the observation of the current environment and the action sequence.

4. A multi-dimensional graphical agent interface integration evaluation method according to claim 1, characterized in that: The S30 includes: S301: Based on the multi-dimensional decomposition quantitative evaluation model, set fine-grained system indicators based on linear temporal logic (LTL) syntax parsing; S302: Based on the fine-grained system indicators, the interface input and output abstract definition layer is run to conduct multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, thus building a multi-dimensional automated evaluation system. Fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators and operational feasibility granularity indicators.

5. A multi-dimensional graphical agent interface integration evaluation method according to claim 1, characterized in that: S40 includes: S401, according to the LTL grammar rules, set a grammar parser to convert the generated text sequence that complies with the LTL grammar rules into an abstract syntax tree; S402: Based on the abstract syntax tree, the LTL formula generated by the agent and visual language large model is parsed to evaluate the structure and nodes, build an evaluation execution layer, and construct an integrated graphical user interface Unify GUI evaluation dataset that includes multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks.

6. A multi-dimensional graphical agent interface integrated evaluation system, characterized in that: include: Multi-dimensional system interface integration subsystem, which integrates graphical agent evaluation interfaces of multi-dimensional systems through standardized processes and input and output representations across operating systems and applications; The multi-dimensional decomposition quantitative evaluation subsystem defines structured input and output using linear temporal logic (LTL) syntax, decomposes the core capabilities of the graphical user interface (GUI) agent into multiple dimensions, and establishes a multi-dimensional decomposition quantitative evaluation model. The sequence matching evaluation system subsystem is based on a multi-dimensional decomposition quantitative evaluation model, sets fine-grained indicators based on linear temporal logic (LTL) syntax parsing, and builds a multi-dimensional automated evaluation system from low-level action sequence matching to high-level task success rate; The cross-platform evaluation dataset subsystem builds the Unify GUI evaluation dataset, which includes multiple cross-platform tasks, based on a multi-dimensional automated evaluation system, and conducts multi-dimensional evaluations of multiple cross-platform tasks.

7. A multi-dimensional graphical agent interface integrated evaluation system according to claim 6, characterized in that: Multi-dimensional system interface integration subsystem, including: The linear temporal language instruction subsystem uses linear temporal logic to implement standardized processes and input and output representations across operating systems and applications, enabling natural language instruction task input and structured output; The multi-dimensional cross-platform interface integration subsystem integrates the graphical agent evaluation interface of the multi-dimensional system; the graphical agent evaluation interface integration of the multi-dimensional system includes: natural language, JSON and Python code integration.

8. The multi-dimensional graphical agent interface integrated evaluation system according to claim 6, characterized in that: Multi-dimensional decomposition quantitative evaluation subsystem, including: The interface agent multi-dimensional decomposition subsystem defines structured input and output through linear temporal logic (LTL) syntax, multi-dimensionally decomposes the core capabilities of the graphical user interface (GUI) agent, and obtains a graphical user interface (GUI) multi-dimensional decomposition module group; The decomposition and quantitative evaluation model subsystem establishes a multi-dimensional decomposition and quantitative evaluation model based on the multi-dimensional decomposition module group of the graphical user interface GUI; The graphical user interface (GUI) multidimensional decomposition module group includes: a task understanding module, a planning generation module, an action positioning module, and a state estimation module; these modules respectively analyze and interpret task instructions in natural language format, expand the target state description into multiple intermediate states, convert the intermediate states into a series of character strings that realize the intermediate states to form an action sequence, and predict the system state after executing the action sequence based on the observation of the current environment and the action sequence.

9. The multi-dimensional graphical agent interface integrated evaluation system according to claim 6, characterized in that: Sequence matching assessment system subsystem, including: The fine-grained system indicator subsystem sets fine-grained system indicators based on linear temporal logic (LTL) syntax parsing based on a multi-dimensional decomposition quantitative evaluation model. The interface operation multi-dimensional evaluation subsystem runs the interface input and output abstract definition layer based on fine-grained system indicators, conducts multi-dimensional automated evaluation from low-level action sequence matching to high-level task success rate, and builds a multi-dimensional automated evaluation system; Fine-grained system indicators include: grammatical correctness granularity indicators, logical consistency granularity indicators and operational feasibility granularity indicators.

10. The multi-dimensional graphical agent interface integrated evaluation system according to claim 6, characterized in that: Cross-platform evaluation dataset subsystem, including: The abstract syntax tree sequence conversion subsystem sets a syntax parser according to the LTL syntax rules and converts the generated text sequence that conforms to the LTL syntax rules into an abstract syntax tree; The syntax parsing cross-platform evaluation subsystem parses the LTL formulas generated by the agent and visual language large model based on the abstract syntax tree, evaluates the structure and nodes, builds the evaluation execution layer, and constructs the Unify GUI evaluation dataset that includes multiple cross-platform tasks to conduct multi-dimensional evaluation of multiple cross-platform tasks.

Images