Model evaluation method, device, equipment and medium

Abstract

The embodiment of the invention relates to a model evaluation method and device, equipment and a medium, and the method comprises the steps: obtaining task description information, and determining a planning task and a first program corresponding to the task description information; converting the planning task into interface calling information corresponding to the first program; inputting the interface calling information into a model, wherein the model is used for operating on a graphical user interface of the first program according to the interface calling information; and obtaining an interface calling log of the model, and determining an evaluation result of the model based on the interface calling log. According to the method, the task is converted into the structured interface calling information, so that the bottom layer difference can be shielded, a standardized action space is provided for decision execution of the model, the evaluation difficulty is reduced, the evaluation value can reflect the comprehensive performance of the model through the evaluation of the multi-dimensional evaluation indexes, and then the accuracy of model evaluation is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a model evaluation method, apparatus, device and medium. Background Technology

[0002] With technological advancements, large models can now perform tasks through interaction with the graphical user interface (GUI) of applications. However, due to the heterogeneity and uncertainty inherent in the interaction with the GUI, large models exhibit poor stability during operation, leading to lower accuracy in their evaluation. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a model evaluation method, apparatus, equipment, and medium.

[0004] This disclosure provides a model evaluation method, the method comprising: Obtain task description information and determine the planning task and first program corresponding to the task description information; The planning task is converted into interface call information corresponding to the first program; The interface call information is input into the model, and the model is used to operate in the graphical user interface of the first program according to the interface call information; Obtain the interface call logs of the model, and determine the evaluation result of the model based on the interface call logs.

[0005] This disclosure also provides a model evaluation apparatus, the apparatus comprising: The acquisition module is used to acquire task description information and determine the planning task and first program corresponding to the task description information; The conversion module is used to convert the planning task into interface call information corresponding to the first program; An operation module is used to input the interface call information into the model, and the model is used to operate in the graphical user interface of the first program according to the interface call information; The evaluation module is used to obtain the interface call logs of the model and determine the evaluation result of the model based on the interface call logs.

[0006] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the executable instructions to implement the model evaluation method provided in this disclosure.

[0007] This disclosure also provides a computer-readable storage medium storing a computer program for executing the model evaluation method provided in this disclosure.

[0008] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: The model evaluation scheme provided in this disclosure obtains task description information, determines the planning task and first program corresponding to the task description information; converts the planning task into interface call information corresponding to the first program; inputs the interface call information into the model, and the model operates according to the interface call information in the graphical user interface of the first program; obtains the model's interface call log, and determines the model's evaluation result based on the interface call log. By adopting the above technical solution, by converting the planning task corresponding to the task description information into the interface call information of the first program to be operated, the model can perform interactive operations in the graphical user interface of the first program according to the interface call information. Then, by obtaining the interface call log based on the interface call information, the model's evaluation result is determined. By converting the task into structured interface call information, underlying differences can be shielded, providing a standardized action space for the model's decision execution, solving the problem of complex and unstable graphical user interface interaction in applications, reducing the evaluation difficulty, and improving the accuracy of model evaluation. Attached Figure Description

[0009] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0010] Figure 1 A schematic flowchart of a model evaluation method provided in an embodiment of this disclosure; Figure 2 A flowchart illustrating another model evaluation method provided in this embodiment of the disclosure; Figure 3 A schematic diagram illustrating a model evaluation and training method provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of the structure of a model evaluation device provided in an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0011] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0012] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0013] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0014] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0015] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0016] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0017] Large models are increasingly used in various applications, especially in professional fields. Their core requirements are to achieve stable interaction with the graphical user interface and efficient task execution. In related technologies, the evaluation methods for models of such graphical user interfaces typically include: 1. Traditional GUI automated testing frameworks, whose specific steps may include identifying GUI elements through locators, writing fixed scripts containing element operations, executing scripts and asserting operation results, and generating test reports containing execution status. This approach has the following limitations: it only focuses on element location and script replay. 2. General Web task benchmark frameworks, whose steps may include providing simple Web tasks such as clicking buttons and filling out forms, recording the model's operation steps and task completion status, and using success rate and number of operation steps as core evaluation indicators. This approach has the following problems: the tasks are mostly short-chain, low-complexity general scenarios, which cannot cover professional application tasks; the evaluation indicators are singular, unable to measure professional quality and economic value, and the evaluation accuracy is low. 3. GUI intelligent agent solutions, which include path self-learning modules, element location modules, exception handling modules, and self-evolutionary systems. While improving operation accuracy by integrating visual models with traditional element picking techniques, achieving a high task success rate, the heterogeneity and uncertainty of professional software GUI interactions lead to poor model operation stability, difficulty in path reproducibility, high evaluation difficulty, and low accuracy.

[0018] To address the aforementioned issues, this disclosure provides a model evaluation method, which will be described below with reference to specific embodiments.

[0019] Figure 1 This is a flowchart illustrating a model evaluation method provided in an embodiment of this disclosure. The method can be executed by a model evaluation device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes: Step 101: Obtain task description information and determine the planned task and first program corresponding to the task description information.

[0020] The model evaluation method of this disclosure can be applied to an evaluation system for evaluating and optimizing models. This system may include a task modeling module, an interface executor, a log recording module, an evaluation module, and a model optimization module. The model can understand natural language instructions and interact with a graphical user interface to complete a series of operations such as clicking and input to achieve user-specified tasks. It can be an intelligent agent that integrates big oracle models and multimodal technologies.

[0021] The task description information can be a task content for a specific application described in natural language by the user. This task description information can be a task description for evaluating the model, such as creating a file in program A. The planning task can be a plan or suggestion of execution steps generated by parsing and processing the task description information. In this embodiment, the planning task includes preconditions, execution actions, and state transition rules. Preconditions can be state conditions that must be met before performing an operation, such as the existence of a compilation error message. Execution actions can be specific operations performed after the preconditions are met, such as calling code modification. State transition rules can be the mapping relationship of how the state changes from the current state to a new state after performing an operation. There can be multiple planning tasks, breaking down complex tasks into multiple planning tasks. The first program can be the application corresponding to the task scenario mapped by the task content of the task description information, such as a drawing program, a document manipulation program, a recording program, etc., and is not specifically limited.

[0022] In this embodiment, the model evaluation device can receive task description information input by the user through the input box of the session interface. The task modeling module analyzes and decomposes the task description information to obtain planned tasks and a first program. The task modeling module can be a pre-processing model for analyzing the task description information. For example, if the task description information is code debugging for program A, four planned tasks can be obtained: opening the code, locating errors, modifying the code, and compiling and verifying. The first program is program A. This structured task decomposition significantly improves the determinism of the evaluation task execution, solving the problem of fuzzy and complex tasks in professional application programs.

[0023] Optionally, the task modeling module can also determine the difficulty value of the task description information. Specifically, this can be achieved by analyzing the number of steps, the number of abnormal branches, and the domain knowledge dependency in the task description information. These factors are then input into a difficulty model to comprehensively determine the corresponding difficulty value. The number of abnormal branches refers to the number of unexpected scenarios that may occur during task execution. Domain knowledge dependency can be the degree of reliance on the professional domain rules or tacit knowledge required by the task. For example, the difficulty value can be calculated as: 0.3 * number of steps + 0.4 * number of abnormal branches + 0.3 * domain knowledge dependency. The weight of the number of steps is 0.3, the weight of the number of abnormal branches is 0.4, and the weight of domain knowledge dependency is 0.3. A number of steps of 1-5 indicates easy, 6-15 indicates medium, and more than 15 indicates complex. This is just an example. Determining the difficulty value of the task description information prepares the model for subsequent selection.

[0024] Step 102: Convert the planning task into the interface call information corresponding to the first program.

[0025] Interface call information can be a complete set of data, parameters, and operation instructions required when the model sends a request to the graphical user interface of the first program to realize interface interaction. Through interface call information, operations can be performed in the graphical user interface environment and the interface status feedback after the operation can be obtained.

[0026] Optionally, the interface call information includes input parameters, output parameters, and exception handling information. Input parameters are those transmitted to the graphical user interface (GUI) environment. These parameters include at least one of the following: interface name, control identifier, and operation coordinates. The interface name can be the operation type corresponding to this GUI call, such as "click" or "input." The control identifier can be a unique identifier for a control within the GUI of the first program, locating a control or element. Operation coordinates are optional. Output parameters define the data returned by the GUI environment. Output parameters can include the execution status and an interface snapshot. The execution status can include success or failure. The interface snapshot can be a screenshot of the interface after an operation is performed in the GUI environment, represented by an interface snapshot identifier and retrieved through the corresponding snapshot address. Exception handling information can include the exception type and corresponding recovery strategy when an operation on the GUI environment encounters an error. For example, it can include six types of exceptions, such as "control not found," and corresponding recovery strategies such as retry and rollback. The interface call information provides a standardized interface for decision execution to the model, shielding underlying differences and improving operational stability.

[0027] The model evaluation device can input the planning task into the interface executor, and the interface executor can convert the planning task into the interface call information of the first program. The interface executor can be a tool that converts the interactive operation of the graphical user interface into the corresponding program with constrained interface call information. The interface executor can be a unified interface call abstraction tool for the graphical user interface interaction settings of the application in this embodiment of the disclosure.

[0028] For example, Figure 2 A flowchart illustrating another model evaluation method provided in this disclosure embodiment is shown below. Figure 2 As shown, in one feasible implementation, the number of planning tasks is multiple, and step 102 may include the following steps: Step 201: Based on the output parameters of the previous planning task, convert the planning task into the interface operation steps corresponding to the first program.

[0029] The interface operation steps can be a description or explanation of the interactive operations corresponding to the planning task in the graphical user interface of the first program. For example, if the planning task is to create a new file in program A, the interface operation steps include moving to a certain coordinate point and clicking on the newly created control.

[0030] After acquiring multiple planning tasks through the interface executor, the model evaluation device can obtain the output parameters of the previous planning task for each planning task. Then, the planning task input instruction parameterization module parses the task intent and determines the specific operation controls in the graphical user interface of the first program based on the interface control layout information of the first program and the output parameters of the previous planning task. The planning task is then converted into interface operation steps. The interface control layout information can be the coordinate information and function information of all controls and elements included in the graphical user interface of the first program. If the planning task is the first planning task, there is no previous planning task, and only the planning task can be analyzed and converted.

[0031] Step 202: Determine the interface call information for the planned task based on the interface operation steps.

[0032] After converting each planning task into the interface operation steps of the first program, the model evaluation device can extract the control identifiers and operation coordinates in the interface operation steps as input parameters for the interface call information, set the output parameters and exception handling information of the interface call information to blank, and fill them in completely after the operation.

[0033] In the above scheme, by setting an interface executor in the evaluation system to perform semantic interface call uniform conversion of the interactive operation of the graphical user interface, the conversion efficiency is guaranteed, and standardized operation instructions for decision execution are provided to the model, thus solving the problem of interaction complexity in heterogeneous graphical user interface environments.

[0034] Step 103: Input the interface call information into the model. The model is used to operate according to the interface call information in the graphical user interface of the first program.

[0035] The model can be used to understand natural language instructions and interact with a graphical user interface (GUI) to complete a series of operations such as clicking and input to achieve user-specified tasks. The GUI can be an interactive object through which the model perceives its environment and performs tasks; it can be an interface system on an electronic device that uses graphical elements to achieve human-computer interaction. Users and the model can exchange information and control operations through graphical elements such as controls and interactive methods such as clicking, swiping, and input. The GUI of the first program can reside within its GUI environment. The GUI environment of the first program can be the complete runtime environment in which the GUI resides, including both the GUI itself and the underlying system and interaction logic supporting its operation.

[0036] In some embodiments, the model is used to perform interactive operations on corresponding controls in the graphical user interface of the first program based on the input parameters in the interface call information, and obtain output parameters. That is, operating the graphical user interface of the first program according to the interface call information through the model may include: performing interactive operations on corresponding controls in the graphical user interface of the first program based on the input parameters in the interface call information through the model, obtaining output parameters, and transmitting the output parameters to the interface executor.

[0037] After the model evaluation device converts the planning task into interface call information corresponding to the first program through the interface executor, it can input this interface call information into the model. The model analyzes and understands the interface call information, and based on the control identifiers and operation coordinates in the input parameters of the interface call information, it interacts with the corresponding controls in the graphical user interface (GUI) of the first program. It obtains the execution status and interface snapshot of the output parameters according to the output parameter format, and feeds the output parameters back to the interface executor. The interface executor can then use the output parameters to convert the interface call information for the next planning task and input it into the model for interaction with the GUI, until the model completes the task and stops, displaying the final output parameters to the user. The model interacts with the program's GUI based on the input parameters in the interface call information and feeds back the output information to the interface executor, improving the accuracy and stability of the graphical user interface interaction.

[0038] Optionally, the model can be determined based on the difficulty value of the task description information. Multiple models can be set to handle tasks with different difficulty levels. By differentiating task difficulty values, models corresponding to the difficulty values ​​can be used to execute tasks, improving the discrimination and accuracy of the evaluation.

[0039] Step 104: Obtain the model's interface call logs and determine the model's evaluation results based on the interface call logs.

[0040] Interface call logs are logs recorded for interactions between the model and the graphical user interface (GUI). These logs record GUI operations based on interface call information, with one log entry per interface call. The evaluation system's logging module deploys event hooks and state detectors between the model and the GUI environment. Optionally, interface call logs may include basic interface call information, execution data, and state snapshots. Basic information may include timestamps, interface names, and parameters (the interface name must match the one in the call information). Execution data may include execution status, execution time, and resource usage. State snapshots may include control tree information and UI snapshots. The format of the interface call logs is unrestricted; for example, they can be stored in JSON format. Interface call logs enable GUI operation replay. Specifically, UI snapshots are associated with snapshot identifiers, and the operations described in the interface call information are reproduced in timestamp order. Breakpoint debugging is also supported.

[0041] Optionally, the evaluation results are the model's evaluation values ​​across multiple evaluation metrics. Evaluation metrics can be standardized measures that quantify model performance. In this embodiment, the evaluation metrics for the model may include multiple metrics, such as success rate, efficiency, robustness, quality, and value. Evaluation values ​​can be numerical values ​​obtained by comprehensively calculating the model across multiple evaluation metrics, and can characterize the model's overall capabilities.

[0042] Specifically, during the process of the model evaluation device operating the interface call information in the graphical user interface of the first program, the interface call log can be recorded by the log recording module. Based on the interface call log, the score of each evaluation indicator can be calculated, and the scores of multiple evaluation indicators can be weighted and summed to obtain the evaluation value.

[0043] For example, assuming multiple evaluation metrics include success rate, efficiency, robustness, quality, and value, the success rate score = number of successfully completed tasks / total number of tasks * 100; the efficiency score = (number of manual steps / number of model steps) * (manual time / model time) * 30 (normalized to 30); the robustness score = number of successful anomaly recovery / total number of anomalies * 20 (normalized to 20); the quality score is determined based on the task description. Taking code tasks as an example, it is calculated through syntax correctness (40%), comment completeness (30%), and naming conventions (30%). The quality score = 0.4 × syntax correctness score + 0.3 × comment completeness score + 0.3 × naming conventions score; the value score = (manual time - model time) * value per hour ÷ 1000 × 20 (normalized to 20). The evaluation value = success rate score + efficiency score + robustness score + quality score + value score, with a maximum score of 100. The higher the evaluation value, the better the model's ability and performance.

[0044] The above solution enables replayability and diagnostics of model operations by recording structured logs of the graphical user interface environment, providing technical support for problem localization during model evaluation and improving the efficiency and accuracy of model evaluation. It also enables the evaluation of the model using multiple evaluation metrics, solving the problem of single evaluation metrics being disconnected from business needs. This allows for precise measurement of the model's capabilities from multiple dimensions, enabling the measurement of the model's comprehensive decision-making ability when executing long-chain and cross-stage tasks, thereby improving the accuracy of the evaluation.

[0045] The model evaluation scheme provided in this disclosure involves: acquiring task description information; determining the planning task and first program corresponding to the task description information; converting the planning task into interface call information corresponding to the first program; inputting the interface call information into the model, which then operates according to the interface call information in the graphical user interface of the first program; acquiring the model's interface call log; and determining the model's evaluation result based on the interface call log. By converting the planning task corresponding to the task description information into interface call information of the first program to be operated, the model can interact with the first program's graphical user interface according to the interface call information. Then, the evaluation result of the model is determined by acquiring the interface call log based on the interface call information. Converting the task into structured interface call information can shield underlying differences, providing a standardized action space for the model's decision-making and execution, solving the problem of complex and unstable graphical user interface interaction in applications, reducing evaluation difficulty, and enabling the evaluation values ​​to reflect the model's comprehensive performance through multi-dimensional evaluation metrics, thereby improving the accuracy of model evaluation.

[0046] In some embodiments, when the evaluation result is the model's evaluation value on multiple evaluation metrics, the model evaluation method may further include: determining a training reward value based on the evaluation value, wherein the training reward value is proportional to the evaluation value; and optimizing the model's parameters based on the training reward value.

[0047] The training reward value serves as a reward signal for training the model. It can be determined based on the evaluation values; the higher the evaluation value, the greater the training reward value. After determining the model's evaluation values ​​on multiple metrics, the model evaluation device can convert these values ​​into training reward values. Specifically, this conversion involves reweighting the scores of multiple evaluation metrics within the evaluation value and performing a weighted calculation. For example, the training reward value can be determined using the following formula: R = α * Success Rate Score + β * Efficiency Score + γ * Robustness Score + δ * Quality Score + ε * Value Score, where R represents the training reward value, and α, β, γ, δ, and ε are weights that can be configured according to the actual situation. These weights can be adjusted based on the importance of the task; for example, they can all be set to 0.2. Furthermore, in some tasks, the efficiency score or robustness score may have the highest weight. The training reward value can then be transmitted to the model optimization module. Based on this training reward value, the corresponding task description information, interface call information, and post-operation output information, the model is trained, and the model parameters are optimized.

[0048] During each training cycle, the model will adjust its parameters based on the training reward value. If an action receives a high training reward value, the model will learn that the action is good and will be more inclined to do similar things in the future. Conversely, if an action receives a low training reward value, the model will avoid doing the action. The model gradually becomes more capable and makes better decisions.

[0049] In the above scheme, the training convergence speed of the model is improved by linking evaluation and training, avoiding the sparse reward problem of traditional reinforcement learning. Compared with the training scheme that only focuses on the task result, the robustness of the model is significantly enhanced.

[0050] For example, Figure 3 This is a schematic diagram illustrating model evaluation and training provided in an embodiment of the present disclosure, such as... Figure 3As shown, taking program A as an example, the process of model evaluation and training by the evaluation system can include: inputting the task description information of program A; analyzing the task through the task modeling module to obtain the planned task, difficulty value, and task scenario, which is the aforementioned program A, and transmitting it to the interaction conversion module; the interaction conversion module includes the control and mapping module, the interface executor, and the instruction parameter module shown in the figure. The control and mapping module outputs the identifier of the task scenario, and the instruction parameterization module parses the task intent, determines the interface name, and transmits it to the interface executor. The interface executor converts the planned task into the interface call information of program A; the model interacts with the graphical user interface of program A through the interface call information; the log recording module records the interaction; the evaluation module calculates the evaluation value by performing five evaluation indicators; the training reward value is calculated; and the model optimization module optimizes the model parameters based on the training reward value. During optimization, the optimization task is also transmitted through the interface executor.

[0051] The technical effects of this solution are as follows: Addressing the issue of unstable graphical user interface (GUI) interaction, it transforms vision-dependent operations into structured interface calls, ensuring that operation paths are recordable and reproducible, thus greatly improving the stability of model interaction with heterogeneous GUI environments. Addressing the issue of single evaluation metrics, it achieves multi-dimensional evaluation from task success to value, quantifying model value through value metrics. Structured task decomposition improves the determinism of evaluation task execution, and the difficulty determination mechanism differentiates the evaluation of models with different capability levels. The model's operation log recording of the GUI improves the efficiency of problem localization during model evaluation, and combined with model optimization, it enhances model iteration efficiency. The closed-loop design of the entire process makes the comprehensive operational capability evaluation of the model closer to real-world scenarios, increasing the proportion of task completion quality conforming to domain standards. The heterogeneous capability collaborative evaluation mechanism for complex business logic jointly evaluates interface interaction operations, code logic understanding, and interface execution strategies, using a weighted scoring system to measure the model's comprehensive business processing level, improving the accuracy and comprehensiveness of the evaluation.

[0052] Figure 4 This is a schematic diagram of a model evaluation device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device. Figure 4 As shown, the device includes: The acquisition module 401 is used to acquire task description information and determine the planning task and the first program corresponding to the task description information; The conversion module 402 is used to convert the planning task into interface call information corresponding to the first program; The operation module 403 is used to input the interface call information into the model, and the model is used to operate in the graphical user interface of the first program according to the interface call information. Evaluation module 404 is used to obtain the interface call logs of the model and determine the evaluation result of the model based on the interface call logs.

[0053] Optionally, module 401 is specifically used for: The task description information is analyzed and decomposed to obtain the planned task and the first program, wherein the planned task includes preconditions, execution actions and state transition rules.

[0054] Optionally, the number of planning tasks is multiple, and the conversion module 402 is used for: Based on the output parameters of the previous planning task, the planning task is converted into the interface operation steps corresponding to the first program; The interface call information for the planned task is determined based on the interface operation steps.

[0055] Optionally, the interface call information includes input parameters, output parameters, and exception handling information. The input parameters include at least one of the interface name, control identifier, and operation coordinates. The output parameters include execution status and interface snapshot.

[0056] Optionally, the model is used to perform interactive operations on corresponding controls in the graphical user interface of the first program based on the input parameters in the interface call information, and obtain output parameters.

[0057] Optionally, the evaluation result is the evaluation value of the model on multiple evaluation indicators, and the device further includes an optimization module, which is used for: A training reward value is determined based on the evaluation value, wherein the training reward value is proportional to the evaluation value; The model parameters are optimized based on the training reward value.

[0058] Optionally, the model is determined based on the difficulty value of the task description information.

[0059] The model evaluation apparatus provided in this disclosure can execute the model evaluation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0060] This disclosure also provides a computer program product, including a computer program / instruction. When executed by a processor, the computer program / instruction implements the model evaluation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0061] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0062] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device 500 in the embodiments of this disclosure. The electronic device 500 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0063] like Figure 5 As shown, the electronic device 500 may include a processing unit 501 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0064] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0065] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined above in the model evaluation method of embodiments of this disclosure.

[0066] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory, a read-only memory, an electrically erasable programmable read-only memory (EPROM), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0067] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol, such as Hypertext Transfer Protocol (HTTP), and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include Local Area Networks (LANs), Wide Area Networks (WANs), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0068] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0069] The aforementioned computer-readable medium carries one or more programs. When the aforementioned one or more programs are executed by the electronic device, the electronic device causes the following: to acquire task description information, determine the planning task and first program corresponding to the task description information; to convert the planning task into interface call information corresponding to the first program; to input the interface call information into a model, the model being used to operate in the graphical user interface of the first program according to the interface call information; and to acquire the interface call log of the model, and determine the evaluation result of the model based on the interface call log.

[0070] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including local area networks (LANs) or wide area networks (WANs), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0071] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0072] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0073] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field-Programmable Gate Array (FPGA), Application Specific Integrated Circuit (ASIC), Application Specific Standard Parts (ASSP), System on Chip (SOC), Complex Programmable Logic Device (CPLD), and so on.

[0074] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, flash memory, optical fiber, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0075] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0076] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0077] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0078] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A model evaluation method, comprising: Obtain task description information and determine the planning task and first program corresponding to the task description information; The planning task is converted into interface call information corresponding to the first program; The interface call information is input into the model, and the model is used to operate in the graphical user interface of the first program according to the interface call information; Obtain the interface call logs of the model, and determine the evaluation result of the model based on the interface call logs.

2. The method according to claim 1, determining the planning task and the first program corresponding to the task description information, includes: The task description information is analyzed and decomposed to obtain the planned task and the first program, wherein the planned task includes preconditions, execution actions and state transition rules.

3. The method according to claim 1, wherein the number of planning tasks is multiple, and converting the planning tasks into interface call information corresponding to the first program includes: Based on the output parameters of the previous planning task, the planning task is converted into the interface operation steps corresponding to the first program; The interface call information for the planned task is determined based on the interface operation steps.

4. The method according to claim 1 or 3, wherein the interface call information includes input parameters, output parameters and exception handling information, wherein the input parameters include at least one of interface name, control identifier and operation coordinates, and the output parameters include execution status and interface snapshot.

5. The method according to claim 4, wherein the model is used to perform interactive operations of corresponding controls in the graphical user interface of the first program based on the input parameters in the interface call information, and obtain output parameters.

6. The method according to claim 1, wherein the evaluation result is the evaluation value of the model on multiple evaluation indicators, and the method further includes: A training reward value is determined based on the evaluation value, wherein the training reward value is proportional to the evaluation value; The parameters of the model are optimized based on the training reward value.

7. The method according to claim 1, wherein the model is determined based on the difficulty value of the task description information.

8. A model evaluation device, comprising: The acquisition module is used to acquire task description information and determine the planning task and first program corresponding to the task description information; The conversion module is used to convert the planning task into interface call information corresponding to the first program; An operation module is used to input the interface call information into the model, and the model is used to operate in the graphical user interface of the first program according to the interface call information; The evaluation module is used to obtain the interface call logs of the model and determine the evaluation result of the model based on the interface call logs.

9. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the model evaluation method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program for performing the model evaluation method according to any one of claims 1-7.

Images